Oct 30 16:20:00.913669 ip-10-0-131-179 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 16:20:00.913678 ip-10-0-131-179 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 16:20:00.913686 ip-10-0-131-179 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 16:20:00.913913 ip-10-0-131-179 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 16:20:11.040060 ip-10-0-131-179 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 16:20:11.040077 ip-10-0-131-179 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot f54f2b897f3740e583edeec2fe0963d7 -- Oct 30 16:22:08.665998 ip-10-0-131-179 systemd[1]: Starting Kubernetes Kubelet... Oct 30 16:22:09.065897 ip-10-0-131-179 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:22:09.065897 ip-10-0-131-179 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 16:22:09.065897 ip-10-0-131-179 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:22:09.065897 ip-10-0-131-179 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:22:09.071479 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071337 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:09.071479 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071478 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:09.071479 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071482 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:09.071479 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071485 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:09.071479 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071488 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071492 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071495 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071497 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071500 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071503 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071505 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071509 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071512 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071515 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071518 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071521 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071523 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071526 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071528 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071532 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071543 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071546 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071549 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:09.071675 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071551 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071554 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071557 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071559 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071562 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071564 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071567 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071570 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071572 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071575 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071578 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071581 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071583 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071586 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071589 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071592 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071594 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071597 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071600 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071602 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:09.072141 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071605 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071607 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071610 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071612 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071616 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071619 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071621 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071623 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071626 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071629 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071631 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071634 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071636 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071639 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071642 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071646 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071650 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071654 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071657 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:09.072606 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071660 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071663 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071665 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071669 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071673 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071676 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071682 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071685 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071688 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071690 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071693 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071695 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071698 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071701 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071703 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071706 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071709 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071711 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071714 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:09.073073 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071716 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071733 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071737 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071740 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071743 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071745 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071748 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071752 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071754 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071757 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071759 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071763 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071766 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071768 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071771 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071773 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071776 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071778 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071781 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071785 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:09.073562 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071788 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071790 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071793 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071796 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.071798 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072617 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072625 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072628 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072631 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072634 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072637 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072640 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072643 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072647 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072649 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072653 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072655 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072658 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072661 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:09.074054 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072665 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072668 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072671 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072673 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072676 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072679 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072681 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072684 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072686 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072689 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072691 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072694 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072697 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072699 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072702 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072705 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072709 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072712 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072715 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:09.074510 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072718 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072738 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072740 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072743 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072746 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072748 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072751 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072753 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072756 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072758 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072762 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072765 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072768 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072771 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072774 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072776 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072778 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072781 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072783 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072786 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:09.074971 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072788 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072790 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072793 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072796 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072798 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072801 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072804 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072806 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072810 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072812 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072815 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072817 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072819 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072822 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072825 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072828 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072830 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072833 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072835 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072838 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:09.075435 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072841 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072843 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072846 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072849 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072852 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072854 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072857 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072859 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072862 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072864 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072866 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072869 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072872 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072874 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072877 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072880 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072882 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072885 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072887 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:09.075962 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072890 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072892 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072896 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072898 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072900 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072903 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072905 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072908 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072910 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072913 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072915 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072918 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072923 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.072927 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073438 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073444 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073448 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073451 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073453 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:09.076426 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073456 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073459 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073461 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073464 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073467 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073470 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073475 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073478 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073480 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073483 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073485 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073488 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073490 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073493 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073495 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073498 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073501 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073504 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073506 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073509 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:09.076890 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073511 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073514 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073517 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073519 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073522 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073525 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073528 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073531 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073534 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073537 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073545 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073549 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073551 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073554 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073556 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073560 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073562 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073565 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073569 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:09.077378 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073572 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073574 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073577 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073580 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073582 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073585 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073588 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073590 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073593 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073596 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073598 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073601 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073603 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073605 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073608 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073610 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073613 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073615 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073618 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073620 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:09.077832 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073623 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073625 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073627 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073630 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073633 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073636 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073639 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073643 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073647 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073650 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073653 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073657 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073659 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073662 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073664 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073667 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073669 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073672 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073674 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:09.078294 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073677 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073679 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073682 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073684 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073686 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073689 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073691 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073694 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073698 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073700 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073703 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073705 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073707 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073710 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073712 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073715 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073717 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073736 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073739 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073742 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:09.078750 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073745 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073748 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.073751 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074912 2574 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074925 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074932 2574 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074936 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074942 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074945 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074956 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074964 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074968 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074971 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074974 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074978 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074981 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074984 2574 flags.go:64] FLAG: --cgroup-root="" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074987 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074990 2574 flags.go:64] FLAG: --client-ca-file="" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074993 2574 flags.go:64] FLAG: --cloud-provider="external" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.074996 2574 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075001 2574 flags.go:64] FLAG: --cluster-domain="" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075004 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075007 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 16:22:09.079210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075027 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075032 2574 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075037 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075040 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075043 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075047 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075050 2574 flags.go:64] FLAG: --contention-profiling="false" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075055 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075058 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075063 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075066 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075070 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075074 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075077 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075080 2574 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075083 2574 flags.go:64] FLAG: --enable-server="true" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075086 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075090 2574 flags.go:64] FLAG: --event-burst="100" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075093 2574 flags.go:64] FLAG: --event-qps="50" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075096 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075099 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075102 2574 flags.go:64] FLAG: --eviction-hard="" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075107 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075110 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075113 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 16:22:09.079791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075116 2574 flags.go:64] FLAG: --eviction-soft="" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075119 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075122 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075125 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075128 2574 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075131 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075134 2574 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075137 2574 flags.go:64] FLAG: --feature-gates="" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075141 2574 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075144 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075147 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075150 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075153 2574 flags.go:64] FLAG: --healthz-port="10248" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075156 2574 flags.go:64] FLAG: --help="false" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075159 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075163 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075168 2574 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075172 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075175 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075179 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075182 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075185 2574 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075187 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 16:22:09.080440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075191 2574 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075194 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075197 2574 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075200 2574 flags.go:64] FLAG: --kube-reserved="" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075203 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075206 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075209 2574 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075211 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075214 2574 flags.go:64] FLAG: --lock-file="" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075217 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075220 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075223 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075228 2574 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075231 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075236 2574 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075239 2574 flags.go:64] FLAG: --logging-format="text" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075242 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075245 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075248 2574 flags.go:64] FLAG: --manifest-url="" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075251 2574 flags.go:64] FLAG: --manifest-url-header="" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075256 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075259 2574 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075263 2574 flags.go:64] FLAG: --max-pods="110" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075266 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075269 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 16:22:09.081009 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075274 2574 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075277 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075280 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075283 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075286 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075294 2574 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075297 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075300 2574 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075303 2574 flags.go:64] FLAG: --pod-cidr="" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075306 2574 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075309 2574 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075312 2574 flags.go:64] FLAG: --pods-per-core="0" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075315 2574 flags.go:64] FLAG: --port="10250" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075318 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075320 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07f6607cfa6e733f5" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075323 2574 flags.go:64] FLAG: --qos-reserved="" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075326 2574 flags.go:64] FLAG: --read-only-port="10255" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075330 2574 flags.go:64] FLAG: --register-node="true" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075332 2574 flags.go:64] FLAG: --register-with-taints="" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075336 2574 flags.go:64] FLAG: --registry-burst="10" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075339 2574 flags.go:64] FLAG: --registry-qps="5" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075343 2574 flags.go:64] FLAG: --reserved-cpus="" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075346 2574 flags.go:64] FLAG: --reserved-memory="" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075350 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075353 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075356 2574 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 16:22:09.081614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075359 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075362 2574 flags.go:64] FLAG: --runonce="false" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075365 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075368 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075371 2574 flags.go:64] FLAG: --seccomp-default="false" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075374 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075378 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075382 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075385 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075388 2574 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075391 2574 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075394 2574 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075397 2574 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075400 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075403 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075406 2574 flags.go:64] FLAG: --system-cgroups="" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075408 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075414 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075416 2574 flags.go:64] FLAG: --tls-cert-file="" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075419 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075423 2574 flags.go:64] FLAG: --tls-min-version="" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075426 2574 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075429 2574 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075432 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075434 2574 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 16:22:09.082271 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075437 2574 flags.go:64] FLAG: --v="2" Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075442 2574 flags.go:64] FLAG: --version="false" Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075448 2574 flags.go:64] FLAG: --vmodule="" Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075452 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075455 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075606 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075611 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075613 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075616 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075619 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075621 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075624 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075626 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075629 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075632 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075635 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075639 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075641 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075644 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075646 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:09.082964 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075649 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075651 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075654 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075657 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075659 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075661 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075664 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075666 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075669 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075671 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075674 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075676 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075679 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075683 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075687 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075690 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075693 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075695 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075697 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:09.083433 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075700 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075702 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075705 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075707 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075710 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075713 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075716 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075733 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075736 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075740 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075743 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075745 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075748 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075750 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075752 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075755 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075757 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075760 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075762 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:09.083877 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075764 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075767 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075769 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075772 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075774 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075777 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075779 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075781 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075785 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075787 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075790 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075792 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075795 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075797 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075800 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075802 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075805 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075808 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075810 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075812 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:09.084357 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075815 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075818 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075820 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075823 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075826 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075828 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075830 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075833 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075835 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075838 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075840 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075843 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075845 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075847 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075850 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075853 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075855 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075857 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075860 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075862 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:09.084956 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075866 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:09.085439 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075868 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:09.085439 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075871 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:09.085439 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075873 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:09.085439 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075875 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:09.085439 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075878 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:09.085439 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075880 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:09.085439 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075883 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:09.085439 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075885 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:09.085439 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075888 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:09.085439 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075890 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:09.085439 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075892 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:09.085439 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075895 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:09.085439 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.075922 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:22:09.085439 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075957 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:09.085439 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075962 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:09.085439 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075965 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075968 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075970 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075973 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075976 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075978 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075981 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075983 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075986 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075989 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075992 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075994 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075996 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.075999 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076001 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076003 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076007 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076009 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076011 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076014 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:09.085825 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076016 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076019 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076021 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076024 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076026 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076028 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076032 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076035 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076038 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076040 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076043 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076045 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076048 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076051 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076054 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076056 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076059 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076061 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076064 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:09.086289 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076066 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076069 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076071 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076074 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076076 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076079 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076081 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076083 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076086 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076088 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076091 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076093 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076095 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076098 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076101 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076103 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076105 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076108 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076110 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076114 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:09.086734 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076116 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076119 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076121 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076124 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076127 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076129 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076132 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076135 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076137 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076139 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076142 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076145 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076147 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076150 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076152 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076155 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076158 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076162 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076170 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:09.087206 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076174 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076176 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076179 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076181 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076184 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076186 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076188 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076191 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076193 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076196 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076200 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076203 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076206 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076209 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076211 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076214 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076216 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076219 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076221 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076224 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:09.087648 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076226 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076229 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076231 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076234 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076236 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.076239 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.083675 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.083691 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.084513 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.084519 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084580 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084585 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084589 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084592 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084595 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084598 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084600 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084603 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084605 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084608 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:09.088121 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084610 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084613 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084615 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084618 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084620 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084623 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084625 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084628 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084630 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084632 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084635 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084637 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084639 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084642 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084644 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084647 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084650 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084653 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084656 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:09.088586 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084669 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084672 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084675 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084678 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084681 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084683 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084686 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084688 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084690 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084693 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084696 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084698 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084701 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084703 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084706 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084709 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084713 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084717 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084736 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:09.089101 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084739 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084742 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084744 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084747 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084750 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084752 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084755 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084757 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084760 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084762 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084765 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084768 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084770 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084773 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084782 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084785 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084788 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084790 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084793 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:09.089613 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084795 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084798 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084800 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084803 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084806 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084808 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084810 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084813 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084816 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084818 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084821 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084823 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084826 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084828 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084831 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084833 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084836 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084839 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084841 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084843 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:09.090120 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084846 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084848 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084851 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084853 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084855 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084857 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084860 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084863 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084871 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084874 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084876 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084879 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084881 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084883 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084886 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084889 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084891 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084893 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:09.090644 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.084896 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:09.091139 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.084924 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:22:09.091139 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085117 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:09.091139 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085121 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:09.091139 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085124 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:09.091139 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085127 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:09.091139 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085129 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:09.091139 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085132 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:09.091139 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085134 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:09.091139 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085137 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:09.091139 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085139 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:09.091139 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085141 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:09.091139 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085144 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:09.091139 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085146 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:09.091139 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085149 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:09.091139 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085151 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085154 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085156 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085159 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085161 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085164 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085166 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085168 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085176 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085180 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085182 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085185 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085187 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085189 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085192 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085195 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085197 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085199 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085202 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:09.091493 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085204 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085207 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085209 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085211 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085214 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085216 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085220 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085223 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085226 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085229 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085232 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085234 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085237 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085239 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085242 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085244 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085246 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085249 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085252 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:09.091989 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085254 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085256 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085259 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085268 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085271 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085273 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085275 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085278 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085280 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085283 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085285 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085288 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085290 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085293 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085295 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085297 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085300 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085302 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085305 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:09.092428 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085307 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085309 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085312 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085314 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085316 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085325 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085327 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085330 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085332 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085343 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085346 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085348 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085351 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085354 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085357 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085359 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085362 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085370 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085373 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085376 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:09.093083 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085378 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:09.093958 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085380 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:09.093958 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085383 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:09.093958 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085386 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:09.093958 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085390 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:09.093958 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085393 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:09.093958 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085396 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:09.093958 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085398 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:09.093958 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085401 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:09.093958 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085403 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:09.093958 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085406 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:09.093958 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085408 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:09.093958 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085411 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:09.093958 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085413 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:09.093958 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085416 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:09.093958 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085418 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:09.093958 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.085442 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085478 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085483 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085486 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085490 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085492 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085495 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085497 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085500 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085503 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085505 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085508 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085510 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085513 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085521 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085523 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085526 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085528 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085531 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085533 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:09.094582 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085536 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085538 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085540 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085543 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085545 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085548 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085550 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085553 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085555 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085557 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085560 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085562 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085564 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085567 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085569 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085572 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085574 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085576 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085579 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:09.095228 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085581 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085584 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085587 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085589 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085591 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085594 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085596 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085599 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085607 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085609 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085612 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085614 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085617 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085619 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085622 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085626 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085629 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085632 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085635 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:09.095688 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085638 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085640 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085643 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085645 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085648 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085650 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085652 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085655 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085658 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085660 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085662 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085665 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085667 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085669 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085672 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085674 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085677 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085681 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085684 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085686 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:09.096423 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085689 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085691 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085699 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085702 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085705 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085707 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085709 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085712 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085714 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085717 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085734 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085736 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085739 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085742 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085744 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085747 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085749 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085752 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085754 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085757 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:09.097023 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085759 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:09.097485 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085762 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:09.097485 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085764 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:09.097485 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085767 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:09.097485 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085769 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:09.097485 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085772 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:09.097485 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085775 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:09.097485 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085778 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:09.097485 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:09.085780 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:09.097485 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.085842 2574 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 16:22:09.097485 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.089322 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 16:22:09.097485 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.090447 2574 server.go:1014] "Starting client certificate rotation" Oct 30 16:22:09.097485 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.090562 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:22:09.097485 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.090612 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:22:09.097485 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.092998 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:22:09.098501 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.098484 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:22:09.109238 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.109221 2574 log.go:25] "Validated CRI v1 runtime API" Oct 30 16:22:09.113142 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.113129 2574 log.go:25] "Validated CRI v1 image API" Oct 30 16:22:09.114023 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.114002 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 16:22:09.116889 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.116869 2574 fs.go:135] Filesystem UUIDs: map[15dd6077-0f08-4130-8124-b9158be1e947:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 acdcf5f9-8141-4a06-8e30-088962adbb5c:/dev/nvme0n1p3] Oct 30 16:22:09.116947 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.116888 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 16:22:09.121200 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.121085 2574 manager.go:217] Machine: {Timestamp:2026-10-30 16:22:09.120593729 +0000 UTC m=+0.355217371 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093315 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2579b3aac5eceb265019f941b958b6 SystemUUID:ec2579b3-aac5-eceb-2650-19f941b958b6 BootID:f54f2b89-7f37-40e5-83ed-eec2fe0963d7 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dc:7a:96:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dc:7a:96:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:88:d0:35:79:de Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 16:22:09.121200 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.121196 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 16:22:09.121312 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.121272 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 16:22:09.125238 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.125216 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 16:22:09.125390 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.125240 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-179.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 16:22:09.125433 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.125396 2574 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 16:22:09.125433 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.125404 2574 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 16:22:09.125433 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.125432 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:22:09.126244 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.126234 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:22:09.127525 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.127515 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 16:22:09.132403 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.132392 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 16:22:09.132533 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.132523 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 16:22:09.133404 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.133393 2574 kubelet.go:493] "Attempting to sync node with API server" Oct 30 16:22:09.133434 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.133409 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 16:22:09.133434 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.133420 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 16:22:09.133434 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.133432 2574 kubelet.go:405] "Adding apiserver pod source" Oct 30 16:22:09.133515 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.133440 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 16:22:09.133898 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.133878 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:22:09.136553 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.136532 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 16:22:09.146488 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.146223 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:22:09.148379 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.147323 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 16:22:09.148379 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.147352 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 16:22:09.148379 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.147495 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 16:22:09.148379 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.147511 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 16:22:09.148379 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.147523 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 16:22:09.148379 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.147529 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 16:22:09.148379 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.147535 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 16:22:09.148379 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.147541 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 16:22:09.149595 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.149580 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 16:22:09.149655 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.149598 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 16:22:09.149655 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.149612 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 16:22:09.149655 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.149622 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 16:22:09.149655 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.149638 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 16:22:09.149655 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.149653 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 16:22:09.150615 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.150603 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 16:22:09.150674 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.150620 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 16:22:09.154733 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.154711 2574 server.go:1263] "Started kubelet" Oct 30 16:22:09.154835 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.154801 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 16:22:09.154893 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.154813 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 16:22:09.154893 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.154878 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 16:22:09.155527 ip-10-0-131-179 systemd[1]: Started Kubernetes Kubelet. Oct 30 16:22:09.156390 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.156377 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 16:22:09.158265 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.158250 2574 server.go:317] "Adding debug handlers to kubelet server" Oct 30 16:22:09.160371 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.160355 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 16:22:09.163528 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.163510 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 16:22:09.164198 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.164185 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 16:22:09.164316 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.164299 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Oct 30 16:22:09.164899 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.164875 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 16:22:09.164899 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.164895 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 16:22:09.165036 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.164878 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 16:22:09.165036 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.165022 2574 reconstruct.go:97] "Volume reconstruction finished" Oct 30 16:22:09.165036 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.165032 2574 reconciler.go:29] "Reconciler: start to sync state" Oct 30 16:22:09.165130 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.165081 2574 factory.go:55] Registering systemd factory Oct 30 16:22:09.165130 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.165101 2574 factory.go:223] Registration of the systemd container factory successfully Oct 30 16:22:09.165390 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.165371 2574 factory.go:153] Registering CRI-O factory Oct 30 16:22:09.165390 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.165383 2574 factory.go:223] Registration of the crio container factory successfully Oct 30 16:22:09.165507 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.165450 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 16:22:09.165507 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.165474 2574 factory.go:103] Registering Raw factory Oct 30 16:22:09.165507 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.165495 2574 manager.go:1199] Started watching for new ooms in manager Oct 30 16:22:09.166804 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.166789 2574 manager.go:319] Starting recovery of all containers Oct 30 16:22:09.178489 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.178470 2574 manager.go:324] Recovery completed Oct 30 16:22:09.180668 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.180648 2574 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 16:22:09.184090 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.184078 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:09.185016 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.184954 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-179.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:22:09.185763 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.185707 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:09.185830 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.185788 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:09.185830 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.185801 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:09.186065 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.184954 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-179.ec2.internal.18e35b2ddc80f21a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-179.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-179.ec2.internal,},FirstTimestamp:2026-10-30 16:22:09.154683418 +0000 UTC m=+0.389307058,LastTimestamp:2026-10-30 16:22:09.154683418 +0000 UTC m=+0.389307058,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-179.ec2.internal,}" Oct 30 16:22:09.186386 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.186372 2574 cpu_manager.go:226] "Starting" policy="none" Oct 30 16:22:09.186386 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.186383 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 16:22:09.186484 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.186388 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:09.186484 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.186399 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 16:22:09.187076 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.187060 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:09.187150 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.187085 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:09.187150 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.187096 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:09.188596 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.188582 2574 policy_none.go:50] "Start" Oct 30 16:22:09.188658 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.188605 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 16:22:09.188658 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.188615 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 16:22:09.189523 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.189502 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 16:22:09.190586 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.190573 2574 policy_none.go:44] "Start" Oct 30 16:22:09.195917 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.195767 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-179.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 16:22:09.213256 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.213235 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-179.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 16:22:09.213370 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.213351 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 16:22:09.213633 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.213473 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-179.ec2.internal.18e35b2dde5b30b7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-179.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-179.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-179.ec2.internal,},FirstTimestamp:2026-10-30 16:22:09.185763511 +0000 UTC m=+0.420387159,LastTimestamp:2026-10-30 16:22:09.185763511 +0000 UTC m=+0.420387159,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-179.ec2.internal,}" Oct 30 16:22:09.213633 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.213601 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 16:22:09.221272 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.221256 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8d886" Oct 30 16:22:09.225273 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.225250 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 16:22:09.227476 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.227465 2574 manager.go:341] "Starting Device Plugin manager" Oct 30 16:22:09.227522 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.227497 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 16:22:09.227522 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.227505 2574 server.go:87] "Starting device plugin registration server" Oct 30 16:22:09.227750 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.227737 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 16:22:09.227821 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.227752 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 16:22:09.227877 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.227835 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 16:22:09.227935 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.227923 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 16:22:09.227993 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.227937 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 16:22:09.228417 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.228402 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 16:22:09.228474 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.228440 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-179.ec2.internal\" not found" Oct 30 16:22:09.234504 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.234445 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-179.ec2.internal.18e35b2dde5bab45 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-179.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-179.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-179.ec2.internal,},FirstTimestamp:2026-10-30 16:22:09.185794885 +0000 UTC m=+0.420418526,LastTimestamp:2026-10-30 16:22:09.185794885 +0000 UTC m=+0.420418526,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-179.ec2.internal,}" Oct 30 16:22:09.235668 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.235652 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8d886" Oct 30 16:22:09.318436 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.318399 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 16:22:09.320537 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.319815 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 16:22:09.320537 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.319836 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 16:22:09.320537 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.319854 2574 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 16:22:09.320537 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.319885 2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 16:22:09.328710 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.328694 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:09.329502 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.329487 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:09.329572 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.329517 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:09.329572 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.329529 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:09.329572 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.329552 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.336064 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.336049 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:09.349644 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.349631 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.349700 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.349652 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-179.ec2.internal\": node \"ip-10-0-131-179.ec2.internal\" not found" Oct 30 16:22:09.389903 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.389880 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Oct 30 16:22:09.420840 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.420821 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal"] Oct 30 16:22:09.420909 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.420879 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:09.421635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.421621 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:09.421689 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.421651 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:09.421689 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.421660 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:09.423908 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.423897 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:09.424053 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.424038 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.424089 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.424070 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:09.424620 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.424602 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:09.424691 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.424630 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:09.424691 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.424645 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:09.424691 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.424651 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:09.424691 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.424668 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:09.424691 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.424680 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:09.426843 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.426829 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.426900 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.426853 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:09.427538 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.427522 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:09.427614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.427549 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:09.427614 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.427564 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:09.452284 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.452270 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-179.ec2.internal\" not found" node="ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.454298 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.454284 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-179.ec2.internal\" not found" node="ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.467609 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.467591 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/26f3b181327243d25a7eb8229eca8f4d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-179.ec2.internal\" (UID: \"26f3b181327243d25a7eb8229eca8f4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.467662 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.467614 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26fcf7a73a4a25dbdf26c9ac58f465b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"26fcf7a73a4a25dbdf26c9ac58f465b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.467662 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.467630 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fcf7a73a4a25dbdf26c9ac58f465b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"26fcf7a73a4a25dbdf26c9ac58f465b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.490336 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.490318 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Oct 30 16:22:09.567985 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.567961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fcf7a73a4a25dbdf26c9ac58f465b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"26fcf7a73a4a25dbdf26c9ac58f465b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.568083 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.567990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/26f3b181327243d25a7eb8229eca8f4d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-179.ec2.internal\" (UID: \"26f3b181327243d25a7eb8229eca8f4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.568083 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.568007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26fcf7a73a4a25dbdf26c9ac58f465b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"26fcf7a73a4a25dbdf26c9ac58f465b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.568083 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.568048 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fcf7a73a4a25dbdf26c9ac58f465b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"26fcf7a73a4a25dbdf26c9ac58f465b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.568083 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.568059 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/26f3b181327243d25a7eb8229eca8f4d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-179.ec2.internal\" (UID: \"26f3b181327243d25a7eb8229eca8f4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.568230 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.568111 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26fcf7a73a4a25dbdf26c9ac58f465b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"26fcf7a73a4a25dbdf26c9ac58f465b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.590621 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.590575 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Oct 30 16:22:09.691220 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.691199 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Oct 30 16:22:09.756012 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.755989 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.757109 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:09.757088 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Oct 30 16:22:09.791996 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.791975 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Oct 30 16:22:09.892514 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.892434 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Oct 30 16:22:09.993014 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:09.992984 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Oct 30 16:22:10.090447 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.090420 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 16:22:10.090937 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.090545 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 16:22:10.093575 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:10.093558 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Oct 30 16:22:10.194307 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:10.194243 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Oct 30 16:22:10.201468 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.201452 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:10.236060 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.236041 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:10.241075 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.241051 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:17:09 +0000 UTC" deadline="2028-08-02 03:20:14.946285118 +0000 UTC" Oct 30 16:22:10.241145 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.241076 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15394h58m4.705212202s" Oct 30 16:22:10.268024 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:10.267995 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fcf7a73a4a25dbdf26c9ac58f465b2.slice/crio-8538278266c801cf13a88f5be3d63f71f57679c2ccdf4d91f7f9eaa542bdcf8b WatchSource:0}: Error finding container 8538278266c801cf13a88f5be3d63f71f57679c2ccdf4d91f7f9eaa542bdcf8b: Status 404 returned error can't find the container with id 8538278266c801cf13a88f5be3d63f71f57679c2ccdf4d91f7f9eaa542bdcf8b Oct 30 16:22:10.268519 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:10.268500 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f3b181327243d25a7eb8229eca8f4d.slice/crio-8e259c703a3d326d9f791f89f8113b7d78f535765ec289b9b8484e171e9b7198 WatchSource:0}: Error finding container 8e259c703a3d326d9f791f89f8113b7d78f535765ec289b9b8484e171e9b7198: Status 404 returned error can't find the container with id 8e259c703a3d326d9f791f89f8113b7d78f535765ec289b9b8484e171e9b7198 Oct 30 16:22:10.274284 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.274267 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:22:10.290678 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.290660 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:10.291613 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.291599 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:10.291712 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.291628 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:10.291712 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.291643 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:10.295172 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:10.295156 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Oct 30 16:22:10.323804 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.323765 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" event={"ID":"26fcf7a73a4a25dbdf26c9ac58f465b2","Type":"ContainerStarted","Data":"8538278266c801cf13a88f5be3d63f71f57679c2ccdf4d91f7f9eaa542bdcf8b"} Oct 30 16:22:10.324759 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.324735 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" event={"ID":"26f3b181327243d25a7eb8229eca8f4d","Type":"ContainerStarted","Data":"8e259c703a3d326d9f791f89f8113b7d78f535765ec289b9b8484e171e9b7198"} Oct 30 16:22:10.396215 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:10.396194 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Oct 30 16:22:10.496714 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:10.496653 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Oct 30 16:22:10.545699 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.545673 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:10.564768 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.564744 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Oct 30 16:22:10.587292 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.587268 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:22:10.588054 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.588036 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:10.623673 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:10.623634 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:22:11.135168 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.135138 2574 apiserver.go:52] "Watching apiserver" Oct 30 16:22:11.164608 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.164579 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 16:22:11.179292 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.179251 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 16:22:11.179649 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.179628 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-wr6tc","openshift-network-diagnostics/network-check-target-hptcz","openshift-ovn-kubernetes/ovnkube-node-l95sm","openshift-image-registry/node-ca-pbsvn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal","openshift-cluster-node-tuning-operator/tuned-rwx47","openshift-multus/network-metrics-daemon-fz7kn","kube-system/global-pull-secret-syncer-pq64k","kube-system/konnectivity-agent-gjhv5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7","openshift-network-operator/iptables-alerter-265mk","openshift-dns/node-resolver-6xn9h","openshift-multus/multus-additional-cni-plugins-7stpt"] Oct 30 16:22:11.182463 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.182442 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.184911 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.184634 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.187078 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.187053 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.189298 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.189274 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:11.193654 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.193632 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.195988 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.195969 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:11.196072 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.196046 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz7kn" podUID="cff25808-bc4e-4bd3-b38c-5503a00d45fa" Oct 30 16:22:11.198268 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.198247 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:11.198366 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.198306 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pq64k" podUID="cb9bd726-ba59-4c55-838f-e00276ca9bef" Oct 30 16:22:11.201490 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.201468 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:11.201590 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.201567 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:11.202260 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.202241 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 16:22:11.202362 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.202351 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5n7vr\"" Oct 30 16:22:11.202478 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.202464 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rjhbz\"" Oct 30 16:22:11.202549 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.202535 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 16:22:11.202635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.202620 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 16:22:11.202821 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.202803 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 16:22:11.202903 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.202866 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 16:22:11.202970 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.202954 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9kf5z\"" Oct 30 16:22:11.203045 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.203032 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 16:22:11.203149 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.203135 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 16:22:11.203649 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.203634 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:11.203827 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.203806 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:11.203912 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.203860 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptcz" podUID="ddae22b7-ddc1-4766-9af6-3899f1edf48c" Oct 30 16:22:11.205468 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.205450 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:22:11.207276 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.207112 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tdd2t\"" Oct 30 16:22:11.207412 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.207320 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 16:22:11.207412 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.207341 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 16:22:11.207525 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.207499 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 16:22:11.207578 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.207548 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 16:22:11.207805 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.207783 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 16:22:11.213785 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.211009 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d6852\"" Oct 30 16:22:11.213785 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.211576 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.213785 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.211970 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 16:22:11.213785 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.212394 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:11.216617 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.216579 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 16:22:11.216699 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.216653 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mzkhm" Oct 30 16:22:11.233850 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.233768 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mzkhm" Oct 30 16:22:11.266692 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.266670 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 16:22:11.275174 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275143 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-kubernetes\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.275299 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275185 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-cni-netd\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.275299 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275214 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovn-node-metrics-cert\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.275299 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275240 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-system-cni-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.275299 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275276 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f9aa963-3cff-41d4-925c-79ba5db70d57-cni-binary-copy\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.275299 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275299 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.275559 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275322 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.275559 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275345 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-cni-bin\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.275559 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275375 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-system-cni-dir\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.275559 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275398 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-cnibin\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.275559 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275425 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64mgb\" (UniqueName: \"kubernetes.io/projected/a7e4f934-2451-4133-aa1a-ee1145ac2c41-kube-api-access-64mgb\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.275559 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275451 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-kubelet\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.275559 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275475 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovnkube-config\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.275920 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275562 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb9bd726-ba59-4c55-838f-e00276ca9bef-dbus\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:11.275920 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275594 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-host-slash\") pod \"iptables-alerter-265mk\" (UID: \"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f\") " pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:11.275920 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275622 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-cni-multus\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.275920 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsjwr\" (UniqueName: \"kubernetes.io/projected/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-kube-api-access-lsjwr\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.275920 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275661 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-systemd\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.275920 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275695 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzxpj\" (UniqueName: \"kubernetes.io/projected/1d0df1a2-2140-46d5-8662-1148296fb5a8-kube-api-access-gzxpj\") pod \"node-resolver-6xn9h\" (UID: \"1d0df1a2-2140-46d5-8662-1148296fb5a8\") " pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:11.275920 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cni-binary-copy\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.275920 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275784 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-cni-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.275920 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-modprobe-d\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.275920 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275874 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-sys\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.276371 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275925 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-host\") pod \"node-ca-pbsvn\" (UID: \"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be\") " pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:11.276371 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.275956 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cnibin\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.276371 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276014 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:11.276371 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-env-overrides\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.276371 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276095 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-iptables-alerter-script\") pod \"iptables-alerter-265mk\" (UID: \"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f\") " pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:11.276371 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276122 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-multus-certs\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.276371 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysctl-d\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.276371 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-slash\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.276371 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276231 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-var-lib-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.276371 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276277 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngkcw\" (UniqueName: \"kubernetes.io/projected/ed9c43e8-604b-44f5-a20c-a3af86848c2c-kube-api-access-ngkcw\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.276371 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276294 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.276371 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276316 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tttc4\" (UniqueName: \"kubernetes.io/projected/58c9bbd6-243d-4769-afc9-19e5e4097ef9-kube-api-access-tttc4\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.276371 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276337 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-tmp\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.276371 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276372 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.277034 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276390 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-systemd\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.277034 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276414 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-serviceca\") pod \"node-ca-pbsvn\" (UID: \"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be\") " pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:11.277034 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-conf-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.277034 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276508 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-sys-fs\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.277034 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276524 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-run-netns\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.277034 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276539 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:11.277034 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276554 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jmvc\" (UniqueName: \"kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc\") pod \"network-check-target-hptcz\" (UID: \"ddae22b7-ddc1-4766-9af6-3899f1edf48c\") " pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:11.277034 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276569 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-kubelet\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.277034 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276583 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysconfig\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.277034 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276597 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysctl-conf\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.277034 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276627 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-host\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.277034 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276655 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-ovn\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.277034 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276683 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt58k\" (UniqueName: \"kubernetes.io/projected/cff25808-bc4e-4bd3-b38c-5503a00d45fa-kube-api-access-rt58k\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:11.277034 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276703 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6089204f-1338-4875-b311-2fa1b27a03d5-konnectivity-ca\") pod \"konnectivity-agent-gjhv5\" (UID: \"6089204f-1338-4875-b311-2fa1b27a03d5\") " pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:22:11.277034 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276748 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-cni-bin\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.277034 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276773 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-daemon-config\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.277671 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276806 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xlkr\" (UniqueName: \"kubernetes.io/projected/6f9aa963-3cff-41d4-925c-79ba5db70d57-kube-api-access-9xlkr\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.277671 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-run\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.277671 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276962 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-lib-modules\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.277671 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.276991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-tuned\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.277671 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277014 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-hostroot\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.277671 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277048 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.277671 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277077 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6089204f-1338-4875-b311-2fa1b27a03d5-agent-certs\") pod \"konnectivity-agent-gjhv5\" (UID: \"6089204f-1338-4875-b311-2fa1b27a03d5\") " pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:22:11.277671 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277102 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-k8s-cni-cncf-io\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.277671 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277128 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-socket-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.277671 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.277671 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277180 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovnkube-script-lib\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.277671 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277206 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d0df1a2-2140-46d5-8662-1148296fb5a8-hosts-file\") pod \"node-resolver-6xn9h\" (UID: \"1d0df1a2-2140-46d5-8662-1148296fb5a8\") " pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:11.277671 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277233 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.277671 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277257 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-etc-kubernetes\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.277671 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277285 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-registration-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.277671 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277311 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-systemd-units\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.278467 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277345 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-log-socket\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.278467 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277372 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9pp4\" (UniqueName: \"kubernetes.io/projected/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-kube-api-access-f9pp4\") pod \"node-ca-pbsvn\" (UID: \"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be\") " pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:11.278467 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277400 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-var-lib-kubelet\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.278467 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277423 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-device-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.278467 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277448 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-node-log\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.278467 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277473 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb9bd726-ba59-4c55-838f-e00276ca9bef-kubelet-config\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:11.278467 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277492 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-etc-selinux\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.278467 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277507 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-etc-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.278467 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277521 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d0df1a2-2140-46d5-8662-1148296fb5a8-tmp-dir\") pod \"node-resolver-6xn9h\" (UID: \"1d0df1a2-2140-46d5-8662-1148296fb5a8\") " pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:11.278467 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277547 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtkhz\" (UniqueName: \"kubernetes.io/projected/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-kube-api-access-vtkhz\") pod \"iptables-alerter-265mk\" (UID: \"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f\") " pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:11.278467 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277566 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-socket-dir-parent\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.278467 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.277599 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-netns\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.293230 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.293208 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:11.298633 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.298610 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lf54z\"" Oct 30 16:22:11.298742 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.298625 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-86hj4\"" Oct 30 16:22:11.298742 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.298694 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 16:22:11.298892 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.298874 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:11.299006 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.298922 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:11.299006 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.298928 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 16:22:11.299006 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.298881 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h4vt2\"" Oct 30 16:22:11.299006 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.298969 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 16:22:11.299006 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.299004 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 16:22:11.299248 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.299008 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 16:22:11.299248 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.299034 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 16:22:11.299248 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.299048 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 16:22:11.299248 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.299098 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 16:22:11.299248 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.299104 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m9x9f\"" Oct 30 16:22:11.299248 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.299149 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 16:22:11.299248 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.299192 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 16:22:11.377981 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.377949 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-cnibin\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.378118 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.377990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64mgb\" (UniqueName: \"kubernetes.io/projected/a7e4f934-2451-4133-aa1a-ee1145ac2c41-kube-api-access-64mgb\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.378118 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378016 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-kubelet\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.378118 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovnkube-config\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.378118 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378062 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb9bd726-ba59-4c55-838f-e00276ca9bef-dbus\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:11.378118 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-host-slash\") pod \"iptables-alerter-265mk\" (UID: \"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f\") " pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:11.378118 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378094 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-cnibin\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.378411 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-cni-multus\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.378411 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378145 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsjwr\" (UniqueName: \"kubernetes.io/projected/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-kube-api-access-lsjwr\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.378411 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-systemd\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.378411 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378193 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzxpj\" (UniqueName: \"kubernetes.io/projected/1d0df1a2-2140-46d5-8662-1148296fb5a8-kube-api-access-gzxpj\") pod \"node-resolver-6xn9h\" (UID: \"1d0df1a2-2140-46d5-8662-1148296fb5a8\") " pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:11.378411 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378218 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cni-binary-copy\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.378411 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378233 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-cni-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.378411 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-modprobe-d\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.378411 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378276 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-sys\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.378411 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-host\") pod \"node-ca-pbsvn\" (UID: \"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be\") " pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:11.378411 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378312 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cnibin\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.378411 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378327 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:11.378411 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378340 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-env-overrides\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.378411 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378354 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-iptables-alerter-script\") pod \"iptables-alerter-265mk\" (UID: \"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f\") " pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:11.378411 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-multus-certs\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.378411 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378386 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysctl-d\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.378411 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378409 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-slash\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378443 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-var-lib-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378459 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkcw\" (UniqueName: \"kubernetes.io/projected/ed9c43e8-604b-44f5-a20c-a3af86848c2c-kube-api-access-ngkcw\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378473 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378488 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tttc4\" (UniqueName: \"kubernetes.io/projected/58c9bbd6-243d-4769-afc9-19e5e4097ef9-kube-api-access-tttc4\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378501 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-tmp\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378521 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378539 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-systemd\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378554 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb9bd726-ba59-4c55-838f-e00276ca9bef-dbus\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378552 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-serviceca\") pod \"node-ca-pbsvn\" (UID: \"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be\") " pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378580 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-conf-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378599 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-host-slash\") pod \"iptables-alerter-265mk\" (UID: \"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f\") " pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378602 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-sys-fs\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378625 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-cni-multus\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-run-netns\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378653 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378659 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-run-netns\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.379155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378669 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmvc\" (UniqueName: \"kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc\") pod \"network-check-target-hptcz\" (UID: \"ddae22b7-ddc1-4766-9af6-3899f1edf48c\") " pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-kubelet\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378688 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-systemd\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysconfig\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378739 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysctl-conf\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378757 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-host\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378790 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-kubelet\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378804 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-host\") pod \"node-ca-pbsvn\" (UID: \"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be\") " pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378835 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysconfig\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378886 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysctl-d\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378906 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-slash\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378927 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-ovn\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378954 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt58k\" (UniqueName: \"kubernetes.io/projected/cff25808-bc4e-4bd3-b38c-5503a00d45fa-kube-api-access-rt58k\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378976 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysctl-conf\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.378986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6089204f-1338-4875-b311-2fa1b27a03d5-konnectivity-ca\") pod \"konnectivity-agent-gjhv5\" (UID: \"6089204f-1338-4875-b311-2fa1b27a03d5\") " pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379013 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-cni-bin\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379005 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-systemd\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.379859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379057 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-daemon-config\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379061 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-cni-bin\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xlkr\" (UniqueName: \"kubernetes.io/projected/6f9aa963-3cff-41d4-925c-79ba5db70d57-kube-api-access-9xlkr\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379103 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-multus-certs\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379109 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-run\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379132 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-lib-modules\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379157 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-host\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379270 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-modprobe-d\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379164 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-tuned\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379306 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-hostroot\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379342 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6089204f-1338-4875-b311-2fa1b27a03d5-agent-certs\") pod \"konnectivity-agent-gjhv5\" (UID: \"6089204f-1338-4875-b311-2fa1b27a03d5\") " pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-k8s-cni-cncf-io\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379369 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovnkube-config\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379404 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-run\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379461 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-socket-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379473 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-cni-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379506 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-k8s-cni-cncf-io\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.380635 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379515 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-lib-modules\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-socket-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379540 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-sys\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379556 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379586 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovnkube-script-lib\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379616 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d0df1a2-2140-46d5-8662-1148296fb5a8-hosts-file\") pod \"node-resolver-6xn9h\" (UID: \"1d0df1a2-2140-46d5-8662-1148296fb5a8\") " pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379646 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379672 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-etc-kubernetes\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379702 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-registration-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379752 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-systemd-units\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379857 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-log-socket\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379777 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-hostroot\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379873 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379891 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9pp4\" (UniqueName: \"kubernetes.io/projected/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-kube-api-access-f9pp4\") pod \"node-ca-pbsvn\" (UID: \"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be\") " pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379923 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-var-lib-kubelet\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379950 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-device-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379975 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-node-log\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.379984 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cni-binary-copy\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.381641 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380006 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb9bd726-ba59-4c55-838f-e00276ca9bef-kubelet-config\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380034 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-etc-selinux\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380059 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-etc-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380084 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d0df1a2-2140-46d5-8662-1148296fb5a8-tmp-dir\") pod \"node-resolver-6xn9h\" (UID: \"1d0df1a2-2140-46d5-8662-1148296fb5a8\") " pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380109 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtkhz\" (UniqueName: \"kubernetes.io/projected/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-kube-api-access-vtkhz\") pod \"iptables-alerter-265mk\" (UID: \"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f\") " pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380131 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-socket-dir-parent\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380154 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380155 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-netns\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380193 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-netns\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380192 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-kubelet\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380243 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-ovn\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380227 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-kubernetes\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380288 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-cni-netd\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380326 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovn-node-metrics-cert\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380339 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-kubernetes\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.379162 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-system-cni-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380430 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-etc-kubernetes\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.382566 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380452 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-system-cni-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.383398 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.380456 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs podName:cff25808-bc4e-4bd3-b38c-5503a00d45fa nodeName:}" failed. No retries permitted until 2026-10-30 16:22:11.880401474 +0000 UTC m=+3.115025122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs") pod "network-metrics-daemon-fz7kn" (UID: "cff25808-bc4e-4bd3-b38c-5503a00d45fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:11.383398 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380476 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-registration-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.383398 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380495 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f9aa963-3cff-41d4-925c-79ba5db70d57-cni-binary-copy\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.383398 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380516 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-var-lib-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.383398 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.383398 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380501 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.383398 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380556 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-log-socket\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.383398 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380598 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-sys-fs\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.383398 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380612 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d0df1a2-2140-46d5-8662-1148296fb5a8-hosts-file\") pod \"node-resolver-6xn9h\" (UID: \"1d0df1a2-2140-46d5-8662-1148296fb5a8\") " pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:11.383398 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380614 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-var-lib-kubelet\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.383398 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380639 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.383398 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380657 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cnibin\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.383398 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380657 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-device-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.383398 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.379394 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:11.383398 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380698 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-node-log\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.383398 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.380742 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret podName:cb9bd726-ba59-4c55-838f-e00276ca9bef nodeName:}" failed. No retries permitted until 2026-10-30 16:22:11.880704125 +0000 UTC m=+3.115327755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret") pod "global-pull-secret-syncer-pq64k" (UID: "cb9bd726-ba59-4c55-838f-e00276ca9bef") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380777 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb9bd726-ba59-4c55-838f-e00276ca9bef-kubelet-config\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380735 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-daemon-config\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380196 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6089204f-1338-4875-b311-2fa1b27a03d5-konnectivity-ca\") pod \"konnectivity-agent-gjhv5\" (UID: \"6089204f-1338-4875-b311-2fa1b27a03d5\") " pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.380560 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.381082 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-env-overrides\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.381260 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.381381 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-etc-selinux\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.381389 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-etc-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.381422 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-cni-netd\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.381573 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-socket-dir-parent\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.381624 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-conf-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.381674 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-systemd-units\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.381860 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-cni-bin\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.381906 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-system-cni-dir\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.381944 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-iptables-alerter-script\") pod \"iptables-alerter-265mk\" (UID: \"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f\") " pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.381968 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f9aa963-3cff-41d4-925c-79ba5db70d57-cni-binary-copy\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.384207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.382013 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.384699 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.382014 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-system-cni-dir\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.384699 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.382046 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.384699 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.382061 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d0df1a2-2140-46d5-8662-1148296fb5a8-tmp-dir\") pod \"node-resolver-6xn9h\" (UID: \"1d0df1a2-2140-46d5-8662-1148296fb5a8\") " pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:11.384699 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.382299 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-cni-bin\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.384699 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.382789 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovnkube-script-lib\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.384699 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.382908 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-serviceca\") pod \"node-ca-pbsvn\" (UID: \"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be\") " pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:11.384699 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.384180 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6089204f-1338-4875-b311-2fa1b27a03d5-agent-certs\") pod \"konnectivity-agent-gjhv5\" (UID: \"6089204f-1338-4875-b311-2fa1b27a03d5\") " pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:22:11.385112 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.385093 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-tmp\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.386062 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.386018 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-tuned\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.386591 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.386574 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovn-node-metrics-cert\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.407354 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.407319 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:22:11.407437 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.407375 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:22:11.407437 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.407391 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2jmvc for pod openshift-network-diagnostics/network-check-target-hptcz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:11.407518 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.407468 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc podName:ddae22b7-ddc1-4766-9af6-3899f1edf48c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:11.907449686 +0000 UTC m=+3.142073334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2jmvc" (UniqueName: "kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc") pod "network-check-target-hptcz" (UID: "ddae22b7-ddc1-4766-9af6-3899f1edf48c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:11.407562 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.407546 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzxpj\" (UniqueName: \"kubernetes.io/projected/1d0df1a2-2140-46d5-8662-1148296fb5a8-kube-api-access-gzxpj\") pod \"node-resolver-6xn9h\" (UID: \"1d0df1a2-2140-46d5-8662-1148296fb5a8\") " pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:11.410012 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.409980 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64mgb\" (UniqueName: \"kubernetes.io/projected/a7e4f934-2451-4133-aa1a-ee1145ac2c41-kube-api-access-64mgb\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.413238 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.413198 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtkhz\" (UniqueName: \"kubernetes.io/projected/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-kube-api-access-vtkhz\") pod \"iptables-alerter-265mk\" (UID: \"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f\") " pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:11.414002 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.413952 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9pp4\" (UniqueName: \"kubernetes.io/projected/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-kube-api-access-f9pp4\") pod \"node-ca-pbsvn\" (UID: \"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be\") " pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:11.414153 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.414129 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsjwr\" (UniqueName: \"kubernetes.io/projected/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-kube-api-access-lsjwr\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.414592 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.414574 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xlkr\" (UniqueName: \"kubernetes.io/projected/6f9aa963-3cff-41d4-925c-79ba5db70d57-kube-api-access-9xlkr\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.415260 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.415239 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt58k\" (UniqueName: \"kubernetes.io/projected/cff25808-bc4e-4bd3-b38c-5503a00d45fa-kube-api-access-rt58k\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:11.421020 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.420998 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngkcw\" (UniqueName: \"kubernetes.io/projected/ed9c43e8-604b-44f5-a20c-a3af86848c2c-kube-api-access-ngkcw\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.421173 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.421088 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tttc4\" (UniqueName: \"kubernetes.io/projected/58c9bbd6-243d-4769-afc9-19e5e4097ef9-kube-api-access-tttc4\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.497324 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.497289 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:11.509037 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:11.509011 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4d6719_619b_46dd_9b1d_93fd5d1e91c0.slice/crio-bab1f009e3083812b75743a0aa793a271586ff2001f420ab1b793fc8233539a7 WatchSource:0}: Error finding container bab1f009e3083812b75743a0aa793a271586ff2001f420ab1b793fc8233539a7: Status 404 returned error can't find the container with id bab1f009e3083812b75743a0aa793a271586ff2001f420ab1b793fc8233539a7 Oct 30 16:22:11.515415 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.515392 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:11.526035 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:11.526004 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e4f934_2451_4133_aa1a_ee1145ac2c41.slice/crio-de4bce0ebcb92f776a9467c258d4736af251ff8a1de64fb34ae08714409dd17f WatchSource:0}: Error finding container de4bce0ebcb92f776a9467c258d4736af251ff8a1de64fb34ae08714409dd17f: Status 404 returned error can't find the container with id de4bce0ebcb92f776a9467c258d4736af251ff8a1de64fb34ae08714409dd17f Oct 30 16:22:11.533792 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.533768 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:11.534816 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.534794 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:11.536974 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.536957 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:11.543778 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:11.543704 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0df1a2_2140_46d5_8662_1148296fb5a8.slice/crio-94e9f3c857ebe0d4f5f59a923374a2f4f52bab39e81de3fa693780477ec79840 WatchSource:0}: Error finding container 94e9f3c857ebe0d4f5f59a923374a2f4f52bab39e81de3fa693780477ec79840: Status 404 returned error can't find the container with id 94e9f3c857ebe0d4f5f59a923374a2f4f52bab39e81de3fa693780477ec79840 Oct 30 16:22:11.548051 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:11.547983 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab0447a_bdef_45aa_92c3_e9d35ee9c2be.slice/crio-cbfd4442672bbde81d226c4cfbbbefbfad195fe1558d970bf1ae6d5f772c4cbc WatchSource:0}: Error finding container cbfd4442672bbde81d226c4cfbbbefbfad195fe1558d970bf1ae6d5f772c4cbc: Status 404 returned error can't find the container with id cbfd4442672bbde81d226c4cfbbbefbfad195fe1558d970bf1ae6d5f772c4cbc Oct 30 16:22:11.551076 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:11.551057 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9c43e8_604b_44f5_a20c_a3af86848c2c.slice/crio-7cb6b0bd449601611dce1c3efc81c137adc82a59ad929bef2538781be2268c14 WatchSource:0}: Error finding container 7cb6b0bd449601611dce1c3efc81c137adc82a59ad929bef2538781be2268c14: Status 404 returned error can't find the container with id 7cb6b0bd449601611dce1c3efc81c137adc82a59ad929bef2538781be2268c14 Oct 30 16:22:11.551170 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.551151 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:22:11.556081 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.556060 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:11.558306 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.558284 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wr6tc" Oct 30 16:22:11.559560 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.559542 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:11.561307 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:11.561286 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6089204f_1338_4875_b311_2fa1b27a03d5.slice/crio-c6646d0f03b00475dcec78da3fdc2ce91bd173d2146c8e0553cfbfd8922b105f WatchSource:0}: Error finding container c6646d0f03b00475dcec78da3fdc2ce91bd173d2146c8e0553cfbfd8922b105f: Status 404 returned error can't find the container with id c6646d0f03b00475dcec78da3fdc2ce91bd173d2146c8e0553cfbfd8922b105f Oct 30 16:22:11.566597 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:11.566572 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c9bbd6_243d_4769_afc9_19e5e4097ef9.slice/crio-ef582cf82ab8be0925f7cfafdf970d224a83c43f40197f79259d1143549e8226 WatchSource:0}: Error finding container ef582cf82ab8be0925f7cfafdf970d224a83c43f40197f79259d1143549e8226: Status 404 returned error can't find the container with id ef582cf82ab8be0925f7cfafdf970d224a83c43f40197f79259d1143549e8226 Oct 30 16:22:11.574553 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:11.574211 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda890f0ad_4cdf_4bb3_b2b5_7ba2caf6802f.slice/crio-3e4c9fa87eb934f94cf97a58acb7e5b132e16fbf5bb86e6cae7cd11327af46f7 WatchSource:0}: Error finding container 3e4c9fa87eb934f94cf97a58acb7e5b132e16fbf5bb86e6cae7cd11327af46f7: Status 404 returned error can't find the container with id 3e4c9fa87eb934f94cf97a58acb7e5b132e16fbf5bb86e6cae7cd11327af46f7 Oct 30 16:22:11.885282 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.884948 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:11.885282 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.885112 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:11.885282 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.885148 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:11.885282 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.885191 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs podName:cff25808-bc4e-4bd3-b38c-5503a00d45fa nodeName:}" failed. No retries permitted until 2026-10-30 16:22:12.885166028 +0000 UTC m=+4.119789673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs") pod "network-metrics-daemon-fz7kn" (UID: "cff25808-bc4e-4bd3-b38c-5503a00d45fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:11.885282 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.885223 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:11.885282 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.885260 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret podName:cb9bd726-ba59-4c55-838f-e00276ca9bef nodeName:}" failed. No retries permitted until 2026-10-30 16:22:12.885249807 +0000 UTC m=+4.119873461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret") pod "global-pull-secret-syncer-pq64k" (UID: "cb9bd726-ba59-4c55-838f-e00276ca9bef") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:11.985737 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:11.985671 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmvc\" (UniqueName: \"kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc\") pod \"network-check-target-hptcz\" (UID: \"ddae22b7-ddc1-4766-9af6-3899f1edf48c\") " pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:11.985921 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.985903 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:22:11.985988 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.985933 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:22:11.985988 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.985946 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2jmvc for pod openshift-network-diagnostics/network-check-target-hptcz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:11.986093 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:11.986005 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc podName:ddae22b7-ddc1-4766-9af6-3899f1edf48c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:12.985988063 +0000 UTC m=+4.220611708 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jmvc" (UniqueName: "kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc") pod "network-check-target-hptcz" (UID: "ddae22b7-ddc1-4766-9af6-3899f1edf48c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:12.237668 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.237508 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:17:11 +0000 UTC" deadline="2028-08-08 12:01:35.938430126 +0000 UTC" Oct 30 16:22:12.237668 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.237573 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15547h39m23.700860898s" Oct 30 16:22:12.351791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.350854 2574 generic.go:356] "Generic (PLEG): container finished" podID="26fcf7a73a4a25dbdf26c9ac58f465b2" containerID="d117de0a8667e68e2ce3cb475691f5c74168e308d5b5fb15d6658378709cfea6" exitCode=0 Oct 30 16:22:12.351791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.350931 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" event={"ID":"26fcf7a73a4a25dbdf26c9ac58f465b2","Type":"ContainerDied","Data":"d117de0a8667e68e2ce3cb475691f5c74168e308d5b5fb15d6658378709cfea6"} Oct 30 16:22:12.359445 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.359364 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gjhv5" event={"ID":"6089204f-1338-4875-b311-2fa1b27a03d5","Type":"ContainerStarted","Data":"c6646d0f03b00475dcec78da3fdc2ce91bd173d2146c8e0553cfbfd8922b105f"} Oct 30 16:22:12.365154 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.365129 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" event={"ID":"ed9c43e8-604b-44f5-a20c-a3af86848c2c","Type":"ContainerStarted","Data":"7cb6b0bd449601611dce1c3efc81c137adc82a59ad929bef2538781be2268c14"} Oct 30 16:22:12.376386 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.376305 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" event={"ID":"a7e4f934-2451-4133-aa1a-ee1145ac2c41","Type":"ContainerStarted","Data":"de4bce0ebcb92f776a9467c258d4736af251ff8a1de64fb34ae08714409dd17f"} Oct 30 16:22:12.415864 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.415758 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rwx47" event={"ID":"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0","Type":"ContainerStarted","Data":"bab1f009e3083812b75743a0aa793a271586ff2001f420ab1b793fc8233539a7"} Oct 30 16:22:12.444140 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.443293 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" event={"ID":"26f3b181327243d25a7eb8229eca8f4d","Type":"ContainerStarted","Data":"fcdf89b54a25e07b2dc3e353b69e8df060d3e81bbe6cd9ab0200b53caa9b1e41"} Oct 30 16:22:12.446313 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.446241 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-265mk" event={"ID":"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f","Type":"ContainerStarted","Data":"3e4c9fa87eb934f94cf97a58acb7e5b132e16fbf5bb86e6cae7cd11327af46f7"} Oct 30 16:22:12.449152 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.449075 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wr6tc" event={"ID":"6f9aa963-3cff-41d4-925c-79ba5db70d57","Type":"ContainerStarted","Data":"63258f8280423ecaf9e3d9bb769a70c1790759e889c97450552d9dcebf05b231"} Oct 30 16:22:12.458879 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.458817 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7stpt" event={"ID":"58c9bbd6-243d-4769-afc9-19e5e4097ef9","Type":"ContainerStarted","Data":"ef582cf82ab8be0925f7cfafdf970d224a83c43f40197f79259d1143549e8226"} Oct 30 16:22:12.467231 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.467183 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pbsvn" event={"ID":"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be","Type":"ContainerStarted","Data":"cbfd4442672bbde81d226c4cfbbbefbfad195fe1558d970bf1ae6d5f772c4cbc"} Oct 30 16:22:12.469646 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.469589 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6xn9h" event={"ID":"1d0df1a2-2140-46d5-8662-1148296fb5a8","Type":"ContainerStarted","Data":"94e9f3c857ebe0d4f5f59a923374a2f4f52bab39e81de3fa693780477ec79840"} Oct 30 16:22:12.894174 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.894140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:12.894311 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.894196 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:12.894382 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:12.894308 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:12.894382 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:12.894368 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret podName:cb9bd726-ba59-4c55-838f-e00276ca9bef nodeName:}" failed. No retries permitted until 2026-10-30 16:22:14.894349922 +0000 UTC m=+6.128973564 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret") pod "global-pull-secret-syncer-pq64k" (UID: "cb9bd726-ba59-4c55-838f-e00276ca9bef") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:12.894794 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:12.894773 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:12.894908 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:12.894830 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs podName:cff25808-bc4e-4bd3-b38c-5503a00d45fa nodeName:}" failed. No retries permitted until 2026-10-30 16:22:14.894813681 +0000 UTC m=+6.129437321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs") pod "network-metrics-daemon-fz7kn" (UID: "cff25808-bc4e-4bd3-b38c-5503a00d45fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:12.994840 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:12.994807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmvc\" (UniqueName: \"kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc\") pod \"network-check-target-hptcz\" (UID: \"ddae22b7-ddc1-4766-9af6-3899f1edf48c\") " pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:12.995004 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:12.994988 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:22:12.995065 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:12.995007 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:22:12.995065 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:12.995020 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2jmvc for pod openshift-network-diagnostics/network-check-target-hptcz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:12.995173 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:12.995090 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc podName:ddae22b7-ddc1-4766-9af6-3899f1edf48c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:14.995074021 +0000 UTC m=+6.229697663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jmvc" (UniqueName: "kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc") pod "network-check-target-hptcz" (UID: "ddae22b7-ddc1-4766-9af6-3899f1edf48c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:13.238195 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:13.238055 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:17:11 +0000 UTC" deadline="2028-04-06 07:10:19.744385037 +0000 UTC" Oct 30 16:22:13.238195 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:13.238115 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12566h48m6.506274077s" Oct 30 16:22:13.320452 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:13.320424 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:13.320627 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:13.320542 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pq64k" podUID="cb9bd726-ba59-4c55-838f-e00276ca9bef" Oct 30 16:22:13.321640 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:13.320816 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:13.321640 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:13.320963 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptcz" podUID="ddae22b7-ddc1-4766-9af6-3899f1edf48c" Oct 30 16:22:13.321640 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:13.321007 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:13.321640 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:13.321071 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz7kn" podUID="cff25808-bc4e-4bd3-b38c-5503a00d45fa" Oct 30 16:22:13.481424 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:13.481389 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" event={"ID":"26fcf7a73a4a25dbdf26c9ac58f465b2","Type":"ContainerStarted","Data":"d9aa510e315096213569a9948c9ae1392c6783b383abd47232d1b7d5ade8361f"} Oct 30 16:22:13.521440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:13.521334 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" podStartSLOduration=3.521314226 podStartE2EDuration="3.521314226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:13.520889962 +0000 UTC m=+4.755513613" watchObservedRunningTime="2026-10-30 16:22:13.521314226 +0000 UTC m=+4.755937876" Oct 30 16:22:13.521609 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:13.521452 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" podStartSLOduration=3.521442609 podStartE2EDuration="3.521442609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:12.487621931 +0000 UTC m=+3.722245583" watchObservedRunningTime="2026-10-30 16:22:13.521442609 +0000 UTC m=+4.756066260" Oct 30 16:22:14.489495 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:14.489463 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7stpt" event={"ID":"58c9bbd6-243d-4769-afc9-19e5e4097ef9","Type":"ContainerStarted","Data":"432586f6b7be6ff757a5bdb7a88159ef8000b9cdf15c5370a62676be257ddfa1"} Oct 30 16:22:14.911360 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:14.911280 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:14.911360 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:14.911337 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:14.911573 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:14.911470 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:14.911573 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:14.911530 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret podName:cb9bd726-ba59-4c55-838f-e00276ca9bef nodeName:}" failed. No retries permitted until 2026-10-30 16:22:18.911513978 +0000 UTC m=+10.146137611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret") pod "global-pull-secret-syncer-pq64k" (UID: "cb9bd726-ba59-4c55-838f-e00276ca9bef") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:14.911943 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:14.911923 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:14.912016 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:14.911979 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs podName:cff25808-bc4e-4bd3-b38c-5503a00d45fa nodeName:}" failed. No retries permitted until 2026-10-30 16:22:18.911965115 +0000 UTC m=+10.146588750 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs") pod "network-metrics-daemon-fz7kn" (UID: "cff25808-bc4e-4bd3-b38c-5503a00d45fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:15.012926 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:15.012319 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmvc\" (UniqueName: \"kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc\") pod \"network-check-target-hptcz\" (UID: \"ddae22b7-ddc1-4766-9af6-3899f1edf48c\") " pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:15.012926 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:15.012491 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:22:15.012926 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:15.012513 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:22:15.012926 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:15.012526 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2jmvc for pod openshift-network-diagnostics/network-check-target-hptcz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:15.012926 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:15.012576 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc podName:ddae22b7-ddc1-4766-9af6-3899f1edf48c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:19.012559358 +0000 UTC m=+10.247182991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jmvc" (UniqueName: "kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc") pod "network-check-target-hptcz" (UID: "ddae22b7-ddc1-4766-9af6-3899f1edf48c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:15.323480 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:15.323399 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:15.323627 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:15.323523 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pq64k" podUID="cb9bd726-ba59-4c55-838f-e00276ca9bef" Oct 30 16:22:15.323974 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:15.323952 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:15.324084 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:15.324056 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptcz" podUID="ddae22b7-ddc1-4766-9af6-3899f1edf48c" Oct 30 16:22:15.324125 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:15.324113 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:15.324201 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:15.324183 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz7kn" podUID="cff25808-bc4e-4bd3-b38c-5503a00d45fa" Oct 30 16:22:15.500524 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:15.498404 2574 generic.go:356] "Generic (PLEG): container finished" podID="58c9bbd6-243d-4769-afc9-19e5e4097ef9" containerID="432586f6b7be6ff757a5bdb7a88159ef8000b9cdf15c5370a62676be257ddfa1" exitCode=0 Oct 30 16:22:15.500524 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:15.498442 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7stpt" event={"ID":"58c9bbd6-243d-4769-afc9-19e5e4097ef9","Type":"ContainerDied","Data":"432586f6b7be6ff757a5bdb7a88159ef8000b9cdf15c5370a62676be257ddfa1"} Oct 30 16:22:16.504046 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:16.503962 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pbsvn" event={"ID":"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be","Type":"ContainerStarted","Data":"f69ee88a86194d69b04a35f1919db2811548e912ecbd8597b73f303f5beffd7e"} Oct 30 16:22:16.509621 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:16.509585 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gjhv5" event={"ID":"6089204f-1338-4875-b311-2fa1b27a03d5","Type":"ContainerStarted","Data":"9ae48906f94735085e97003ad7ae0df4d758fa109dc6f2ae30f4dea7395bb934"} Oct 30 16:22:16.584615 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:16.584556 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pbsvn" podStartSLOduration=3.073678725 podStartE2EDuration="7.584534648s" totalImagesPullingTime="4.510855923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:16.544481593 +0000 UTC m=+7.779105242" watchObservedRunningTime="2026-10-30 16:22:16.584534648 +0000 UTC m=+7.819158299" Oct 30 16:22:17.326587 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:17.325861 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:17.326587 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:17.325870 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:17.326587 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:17.325976 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pq64k" podUID="cb9bd726-ba59-4c55-838f-e00276ca9bef" Oct 30 16:22:17.326587 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:17.325997 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:17.326587 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:17.326078 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz7kn" podUID="cff25808-bc4e-4bd3-b38c-5503a00d45fa" Oct 30 16:22:17.326587 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:17.326127 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptcz" podUID="ddae22b7-ddc1-4766-9af6-3899f1edf48c" Oct 30 16:22:17.516410 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:17.516332 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" event={"ID":"a7e4f934-2451-4133-aa1a-ee1145ac2c41","Type":"ContainerStarted","Data":"bd6ee1c54fcf3f4764afc8f28751ab815e6abec99ab98b67634ad82a7c1680f1"} Oct 30 16:22:18.728083 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:18.728042 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 16:22:18.948359 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:18.948313 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:18.948498 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:18.948382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:18.948559 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:18.948537 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:18.948636 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:18.948603 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret podName:cb9bd726-ba59-4c55-838f-e00276ca9bef nodeName:}" failed. No retries permitted until 2026-10-30 16:22:26.94858542 +0000 UTC m=+18.183209048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret") pod "global-pull-secret-syncer-pq64k" (UID: "cb9bd726-ba59-4c55-838f-e00276ca9bef") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:18.949055 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:18.949034 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:18.949133 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:18.949097 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs podName:cff25808-bc4e-4bd3-b38c-5503a00d45fa nodeName:}" failed. No retries permitted until 2026-10-30 16:22:26.949082085 +0000 UTC m=+18.183705719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs") pod "network-metrics-daemon-fz7kn" (UID: "cff25808-bc4e-4bd3-b38c-5503a00d45fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:19.049672 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:19.049636 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmvc\" (UniqueName: \"kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc\") pod \"network-check-target-hptcz\" (UID: \"ddae22b7-ddc1-4766-9af6-3899f1edf48c\") " pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:19.049875 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:19.049857 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:22:19.049928 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:19.049885 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:22:19.049928 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:19.049899 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2jmvc for pod openshift-network-diagnostics/network-check-target-hptcz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:19.050018 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:19.049954 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc podName:ddae22b7-ddc1-4766-9af6-3899f1edf48c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:27.049938087 +0000 UTC m=+18.284561730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jmvc" (UniqueName: "kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc") pod "network-check-target-hptcz" (UID: "ddae22b7-ddc1-4766-9af6-3899f1edf48c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:19.233354 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:19.233244 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:22:18.728065268Z","UUID":"49ad1854-4ec4-41dd-9365-4e26d48806aa","Handler":null,"Name":"","Endpoint":""} Oct 30 16:22:19.234966 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:19.234935 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 16:22:19.235081 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:19.234975 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 16:22:19.253444 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:19.253425 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:22:19.325312 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:19.325248 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:19.325452 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:19.325253 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:19.325452 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:19.325378 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz7kn" podUID="cff25808-bc4e-4bd3-b38c-5503a00d45fa" Oct 30 16:22:19.325452 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:19.325390 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:19.325618 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:19.325465 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptcz" podUID="ddae22b7-ddc1-4766-9af6-3899f1edf48c" Oct 30 16:22:19.325618 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:19.325574 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pq64k" podUID="cb9bd726-ba59-4c55-838f-e00276ca9bef" Oct 30 16:22:19.524678 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:19.524439 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" event={"ID":"a7e4f934-2451-4133-aa1a-ee1145ac2c41","Type":"ContainerStarted","Data":"b2264d2b32e9ac9247d659db35b5dc07135befccd2c80eb7cc5315704ec5643e"} Oct 30 16:22:19.527469 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:19.526942 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wr6tc" event={"ID":"6f9aa963-3cff-41d4-925c-79ba5db70d57","Type":"ContainerStarted","Data":"578f1892e5a38e1c68f141f4a48256d0aec11f28d521615238696e9bd0e0721c"} Oct 30 16:22:19.528833 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:19.528806 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6xn9h" event={"ID":"1d0df1a2-2140-46d5-8662-1148296fb5a8","Type":"ContainerStarted","Data":"2a0024d0f3398750b9a7b8c8f8a01bb52141922d64f99877bb36777ca46865da"} Oct 30 16:22:19.570462 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:19.569665 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gjhv5" podStartSLOduration=6.656492513 podStartE2EDuration="10.569653136s" totalImagesPullingTime="3.913160623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:16.584972423 +0000 UTC m=+7.819596074" watchObservedRunningTime="2026-10-30 16:22:19.569653136 +0000 UTC m=+10.804276789" Oct 30 16:22:19.570462 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:19.570031 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-wr6tc" podStartSLOduration=3.322913857 podStartE2EDuration="10.570020469s" totalImagesPullingTime="7.247106612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:19.569004452 +0000 UTC m=+10.803628101" watchObservedRunningTime="2026-10-30 16:22:19.570020469 +0000 UTC m=+10.804644119" Oct 30 16:22:19.607587 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:19.607529 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6xn9h" podStartSLOduration=2.919792471 podStartE2EDuration="10.607509854s" totalImagesPullingTime="7.687717383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:19.607227816 +0000 UTC m=+10.841851467" watchObservedRunningTime="2026-10-30 16:22:19.607509854 +0000 UTC m=+10.842133506" Oct 30 16:22:19.818555 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:19.818522 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:22:19.819388 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:19.819369 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:22:20.533286 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:20.533253 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-265mk" event={"ID":"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f","Type":"ContainerStarted","Data":"fbe2936a1ca5a9b6abea40296ccf37b3c4d4ecc562c309d747b3d6387ccfc3e3"} Oct 30 16:22:20.535647 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:20.535620 2574 generic.go:356] "Generic (PLEG): container finished" podID="58c9bbd6-243d-4769-afc9-19e5e4097ef9" containerID="cefcace71706c7a063c687137b0ead8c2a7e78eed98ae2040cd6ee95d4d35dfd" exitCode=0 Oct 30 16:22:20.535805 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:20.535698 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7stpt" event={"ID":"58c9bbd6-243d-4769-afc9-19e5e4097ef9","Type":"ContainerDied","Data":"cefcace71706c7a063c687137b0ead8c2a7e78eed98ae2040cd6ee95d4d35dfd"} Oct 30 16:22:20.539813 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:20.539788 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" event={"ID":"a7e4f934-2451-4133-aa1a-ee1145ac2c41","Type":"ContainerStarted","Data":"c860b14c5b5981fd703d311767b826efb42a9f9999c17502041accb63d79bb92"} Oct 30 16:22:20.540557 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:20.540539 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:22:20.573058 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:20.573009 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-265mk" podStartSLOduration=4.116781189 podStartE2EDuration="11.572995611s" totalImagesPullingTime="7.456214422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:20.572503004 +0000 UTC m=+11.807126655" watchObservedRunningTime="2026-10-30 16:22:20.572995611 +0000 UTC m=+11.807619262" Oct 30 16:22:20.657622 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:20.657571 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" podStartSLOduration=3.692161157 podStartE2EDuration="11.65755721s" totalImagesPullingTime="7.965396053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:20.614213853 +0000 UTC m=+11.848837503" watchObservedRunningTime="2026-10-30 16:22:20.65755721 +0000 UTC m=+11.892180861" Oct 30 16:22:21.320606 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:21.320533 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:21.321145 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:21.320533 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:21.321145 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:21.320670 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:21.321145 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:21.320666 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptcz" podUID="ddae22b7-ddc1-4766-9af6-3899f1edf48c" Oct 30 16:22:21.321145 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:21.320778 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pq64k" podUID="cb9bd726-ba59-4c55-838f-e00276ca9bef" Oct 30 16:22:21.321145 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:21.320860 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz7kn" podUID="cff25808-bc4e-4bd3-b38c-5503a00d45fa" Oct 30 16:22:21.543464 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:21.543422 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rwx47" event={"ID":"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0","Type":"ContainerStarted","Data":"7ab23986bbefad4c7a2a191ac38aeb86e14a0351889bd043663c5f2df6b26f73"} Oct 30 16:22:21.546279 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:21.546247 2574 generic.go:356] "Generic (PLEG): container finished" podID="58c9bbd6-243d-4769-afc9-19e5e4097ef9" containerID="92271daeb0ef2812da5c46ab810a578bb74fa352a8bd7f18723595b249bdcc6a" exitCode=0 Oct 30 16:22:21.546419 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:21.546366 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7stpt" event={"ID":"58c9bbd6-243d-4769-afc9-19e5e4097ef9","Type":"ContainerDied","Data":"92271daeb0ef2812da5c46ab810a578bb74fa352a8bd7f18723595b249bdcc6a"} Oct 30 16:22:21.584797 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:21.584692 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rwx47" podStartSLOduration=2.741679431 podStartE2EDuration="12.584675689s" totalImagesPullingTime="9.842996258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:21.584222658 +0000 UTC m=+12.818846309" watchObservedRunningTime="2026-10-30 16:22:21.584675689 +0000 UTC m=+12.819299342" Oct 30 16:22:23.321022 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:23.320819 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:23.321674 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:23.320852 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:23.321674 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:23.321104 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptcz" podUID="ddae22b7-ddc1-4766-9af6-3899f1edf48c" Oct 30 16:22:23.321674 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:23.320921 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:23.321674 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:23.321192 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pq64k" podUID="cb9bd726-ba59-4c55-838f-e00276ca9bef" Oct 30 16:22:23.321674 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:23.321297 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz7kn" podUID="cff25808-bc4e-4bd3-b38c-5503a00d45fa" Oct 30 16:22:23.554256 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:23.554223 2574 generic.go:356] "Generic (PLEG): container finished" podID="58c9bbd6-243d-4769-afc9-19e5e4097ef9" containerID="a526b901f5a420caed94ddb69934b7708dc148cdc4f21e92c388af2fc13f8e3c" exitCode=0 Oct 30 16:22:23.554425 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:23.554283 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7stpt" event={"ID":"58c9bbd6-243d-4769-afc9-19e5e4097ef9","Type":"ContainerDied","Data":"a526b901f5a420caed94ddb69934b7708dc148cdc4f21e92c388af2fc13f8e3c"} Oct 30 16:22:25.323672 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:25.323635 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:25.324151 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:25.323734 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:25.324151 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:25.323846 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:25.324151 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:25.323861 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz7kn" podUID="cff25808-bc4e-4bd3-b38c-5503a00d45fa" Oct 30 16:22:25.324151 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:25.323924 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pq64k" podUID="cb9bd726-ba59-4c55-838f-e00276ca9bef" Oct 30 16:22:25.324151 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:25.323997 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptcz" podUID="ddae22b7-ddc1-4766-9af6-3899f1edf48c" Oct 30 16:22:26.564629 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:26.564436 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" event={"ID":"ed9c43e8-604b-44f5-a20c-a3af86848c2c","Type":"ContainerStarted","Data":"79a9bddfe7c4a7a62d623dd932be2d24333131b22ad550987c92bce56b450ada"} Oct 30 16:22:26.565081 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:26.564682 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" event={"ID":"ed9c43e8-604b-44f5-a20c-a3af86848c2c","Type":"ContainerStarted","Data":"b271d0a1bf033d6e60b5cb16dbecea41bda8df2afa33a40cf0ec4e737366c29a"} Oct 30 16:22:26.565081 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:26.564734 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" event={"ID":"ed9c43e8-604b-44f5-a20c-a3af86848c2c","Type":"ContainerStarted","Data":"0cbe7a3f734b754f2ff9a13693d6a995efba8b1ed7241e858709669c4e8856ad"} Oct 30 16:22:26.565081 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:26.564781 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" event={"ID":"ed9c43e8-604b-44f5-a20c-a3af86848c2c","Type":"ContainerStarted","Data":"76bc0f16e19bc04f07d11866bd7b5a6a8a80c1d3d6df765353dfd3e3d9ed1c39"} Oct 30 16:22:26.565081 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:26.564811 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" event={"ID":"ed9c43e8-604b-44f5-a20c-a3af86848c2c","Type":"ContainerStarted","Data":"e2ec7746d64c3b97a09444947f6f1da1812dc151a1e2fac1febb84ec15c3ea49"} Oct 30 16:22:26.565081 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:26.564885 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" event={"ID":"ed9c43e8-604b-44f5-a20c-a3af86848c2c","Type":"ContainerStarted","Data":"a8f2bf25ce0a4462314e82fd178d35805cf067f3cb55c1ec37fbb86ac905223f"} Oct 30 16:22:26.567510 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:26.567486 2574 generic.go:356] "Generic (PLEG): container finished" podID="58c9bbd6-243d-4769-afc9-19e5e4097ef9" containerID="104331adf15444592c34876d6dd4e0a0e7f579796eb5b34de0fcbcfd0ad959e4" exitCode=0 Oct 30 16:22:26.567620 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:26.567521 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7stpt" event={"ID":"58c9bbd6-243d-4769-afc9-19e5e4097ef9","Type":"ContainerDied","Data":"104331adf15444592c34876d6dd4e0a0e7f579796eb5b34de0fcbcfd0ad959e4"} Oct 30 16:22:27.012712 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:27.012680 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:27.012712 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:27.012718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:27.012909 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:27.012831 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:27.012909 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:27.012839 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:27.012909 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:27.012875 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret podName:cb9bd726-ba59-4c55-838f-e00276ca9bef nodeName:}" failed. No retries permitted until 2026-10-30 16:22:43.012864251 +0000 UTC m=+34.247487879 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret") pod "global-pull-secret-syncer-pq64k" (UID: "cb9bd726-ba59-4c55-838f-e00276ca9bef") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:22:27.012909 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:27.012890 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs podName:cff25808-bc4e-4bd3-b38c-5503a00d45fa nodeName:}" failed. No retries permitted until 2026-10-30 16:22:43.012883321 +0000 UTC m=+34.247506949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs") pod "network-metrics-daemon-fz7kn" (UID: "cff25808-bc4e-4bd3-b38c-5503a00d45fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:22:27.113038 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:27.113007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmvc\" (UniqueName: \"kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc\") pod \"network-check-target-hptcz\" (UID: \"ddae22b7-ddc1-4766-9af6-3899f1edf48c\") " pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:27.113183 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:27.113115 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:22:27.113183 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:27.113131 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:22:27.113183 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:27.113140 2574 projected.go:196] Error preparing data for projected volume kube-api-access-2jmvc for pod openshift-network-diagnostics/network-check-target-hptcz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:27.113285 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:27.113185 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc podName:ddae22b7-ddc1-4766-9af6-3899f1edf48c nodeName:}" failed. No retries permitted until 2026-10-30 16:22:43.11317309 +0000 UTC m=+34.347796731 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jmvc" (UniqueName: "kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc") pod "network-check-target-hptcz" (UID: "ddae22b7-ddc1-4766-9af6-3899f1edf48c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:22:27.320132 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:27.320057 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:27.320132 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:27.320091 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:27.320300 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:27.320155 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pq64k" podUID="cb9bd726-ba59-4c55-838f-e00276ca9bef" Oct 30 16:22:27.320300 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:27.320216 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:27.320356 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:27.320340 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz7kn" podUID="cff25808-bc4e-4bd3-b38c-5503a00d45fa" Oct 30 16:22:27.320471 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:27.320445 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptcz" podUID="ddae22b7-ddc1-4766-9af6-3899f1edf48c" Oct 30 16:22:27.572940 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:27.572855 2574 generic.go:356] "Generic (PLEG): container finished" podID="58c9bbd6-243d-4769-afc9-19e5e4097ef9" containerID="05ba1822c46a34496edf469090f4762df1503e228f85580a80e0a8af1bd42cfa" exitCode=0 Oct 30 16:22:27.572940 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:27.572920 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7stpt" event={"ID":"58c9bbd6-243d-4769-afc9-19e5e4097ef9","Type":"ContainerDied","Data":"05ba1822c46a34496edf469090f4762df1503e228f85580a80e0a8af1bd42cfa"} Oct 30 16:22:28.579381 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:28.579358 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" event={"ID":"ed9c43e8-604b-44f5-a20c-a3af86848c2c","Type":"ContainerStarted","Data":"3a411bd87c9c556692cc8b43f9992b8699505c5d04a0c3361b7a2a81de2ab73c"} Oct 30 16:22:28.583335 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:28.583308 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7stpt" event={"ID":"58c9bbd6-243d-4769-afc9-19e5e4097ef9","Type":"ContainerStarted","Data":"7b292850dfcc3a900aef4d6327bc2e29f1ca58cca5244f060b0885cd90642584"} Oct 30 16:22:28.654276 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:28.654230 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7stpt" podStartSLOduration=9.57887957 podStartE2EDuration="19.654217598s" totalImagesPullingTime="10.075338028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:28.640311862 +0000 UTC m=+19.874935512" watchObservedRunningTime="2026-10-30 16:22:28.654217598 +0000 UTC m=+19.888841253" Oct 30 16:22:29.321218 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:29.321188 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:29.321360 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:29.321267 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptcz" podUID="ddae22b7-ddc1-4766-9af6-3899f1edf48c" Oct 30 16:22:29.321360 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:29.321347 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:29.321507 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:29.321443 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz7kn" podUID="cff25808-bc4e-4bd3-b38c-5503a00d45fa" Oct 30 16:22:29.321789 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:29.321766 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:29.322128 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:29.322084 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pq64k" podUID="cb9bd726-ba59-4c55-838f-e00276ca9bef" Oct 30 16:22:31.321260 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:31.321070 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:31.321942 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:31.321077 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:31.321942 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:31.321335 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptcz" podUID="ddae22b7-ddc1-4766-9af6-3899f1edf48c" Oct 30 16:22:31.321942 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:31.321427 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pq64k" podUID="cb9bd726-ba59-4c55-838f-e00276ca9bef" Oct 30 16:22:31.321942 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:31.321110 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:31.321942 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:31.321539 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz7kn" podUID="cff25808-bc4e-4bd3-b38c-5503a00d45fa" Oct 30 16:22:31.593509 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:31.593432 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" event={"ID":"ed9c43e8-604b-44f5-a20c-a3af86848c2c","Type":"ContainerStarted","Data":"4a9bedc59dd0e5f2703819e32bda645d7a62eda5c32867de0a598b56a39a69da"} Oct 30 16:22:31.593779 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:31.593759 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:31.610529 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:31.610507 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:31.680547 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:31.680505 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" podStartSLOduration=8.043434717 podStartE2EDuration="22.680494828s" totalImagesPullingTime="14.637060111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:31.666892545 +0000 UTC m=+22.901516196" watchObservedRunningTime="2026-10-30 16:22:31.680494828 +0000 UTC m=+22.915118482" Oct 30 16:22:32.596036 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:32.596011 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:22:32.596534 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:32.596508 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:32.614664 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:32.614634 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:32.714509 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:32.714481 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fz7kn"] Oct 30 16:22:32.714646 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:32.714587 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:32.714687 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:32.714672 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz7kn" podUID="cff25808-bc4e-4bd3-b38c-5503a00d45fa" Oct 30 16:22:32.717459 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:32.717433 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pq64k"] Oct 30 16:22:32.717575 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:32.717538 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:32.717642 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:32.717625 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pq64k" podUID="cb9bd726-ba59-4c55-838f-e00276ca9bef" Oct 30 16:22:32.717944 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:32.717927 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hptcz"] Oct 30 16:22:32.718023 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:32.718012 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:32.718110 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:32.718089 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptcz" podUID="ddae22b7-ddc1-4766-9af6-3899f1edf48c" Oct 30 16:22:33.597851 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:33.597823 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:22:34.320229 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:34.320200 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:34.320384 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:34.320200 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:34.320384 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:34.320301 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz7kn" podUID="cff25808-bc4e-4bd3-b38c-5503a00d45fa" Oct 30 16:22:34.320384 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:34.320204 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:34.320477 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:34.320385 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pq64k" podUID="cb9bd726-ba59-4c55-838f-e00276ca9bef" Oct 30 16:22:34.320477 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:34.320440 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptcz" podUID="ddae22b7-ddc1-4766-9af6-3899f1edf48c" Oct 30 16:22:34.600184 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:34.600097 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:22:35.624180 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:35.624153 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:35.624623 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:35.624323 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:22:35.635215 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:35.635181 2574 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" podUID="ed9c43e8-604b-44f5-a20c-a3af86848c2c" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 16:22:35.643903 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:35.643869 2574 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" podUID="ed9c43e8-604b-44f5-a20c-a3af86848c2c" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 16:22:36.320934 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:36.320865 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:36.320934 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:36.320900 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:36.321095 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:36.320971 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz7kn" podUID="cff25808-bc4e-4bd3-b38c-5503a00d45fa" Oct 30 16:22:36.321095 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:36.321009 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:36.321095 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:36.321077 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pq64k" podUID="cb9bd726-ba59-4c55-838f-e00276ca9bef" Oct 30 16:22:36.321184 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:36.321163 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptcz" podUID="ddae22b7-ddc1-4766-9af6-3899f1edf48c" Oct 30 16:22:38.320264 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.320238 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:38.320616 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.320321 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:38.320616 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.320335 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:38.320616 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.320412 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hptcz" podUID="ddae22b7-ddc1-4766-9af6-3899f1edf48c" Oct 30 16:22:38.320616 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.320498 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz7kn" podUID="cff25808-bc4e-4bd3-b38c-5503a00d45fa" Oct 30 16:22:38.320616 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.320590 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pq64k" podUID="cb9bd726-ba59-4c55-838f-e00276ca9bef" Oct 30 16:22:38.605705 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.605673 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeReady" Oct 30 16:22:38.605885 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.605872 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 16:22:38.658025 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.658001 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5r74t"] Oct 30 16:22:38.661388 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.661369 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5r74t" Oct 30 16:22:38.661525 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.661389 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9b9bn"] Oct 30 16:22:38.664333 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.664314 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2sc7w"] Oct 30 16:22:38.664451 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.664415 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9b9bn" Oct 30 16:22:38.667767 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.667066 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8"] Oct 30 16:22:38.667767 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.667573 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2sc7w" Oct 30 16:22:38.669953 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.669932 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp"] Oct 30 16:22:38.670072 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.670056 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.672794 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.672774 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-58c95ddf47-6m2jw"] Oct 30 16:22:38.672913 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.672882 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:38.675604 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.675585 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rmgmf"] Oct 30 16:22:38.675711 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.675702 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:38.679817 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.679797 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm"] Oct 30 16:22:38.680617 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.680591 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rmgmf" Oct 30 16:22:38.681610 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.681589 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 16:22:38.688605 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.688583 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 16:22:38.688707 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.688674 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 16:22:38.688809 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.688797 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r5btp\"" Oct 30 16:22:38.688961 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.688946 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 16:22:38.690304 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.690286 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dfdc4b7f-lkk6g"] Oct 30 16:22:38.690403 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.690374 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:38.691848 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.691826 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 16:22:38.691938 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.691924 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pzcq9\"" Oct 30 16:22:38.693168 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.693150 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kx55q"] Oct 30 16:22:38.693272 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.693259 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.696118 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.696098 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44"] Oct 30 16:22:38.696474 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.696456 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.700133 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.700116 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-szx5s"] Oct 30 16:22:38.700317 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.700228 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:38.704578 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.704553 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf"] Oct 30 16:22:38.706607 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.706585 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:38.707706 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.707690 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-sj6h8"] Oct 30 16:22:38.707831 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.707802 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:38.710815 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.710620 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6xx9g"] Oct 30 16:22:38.710815 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.710795 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sj6h8" Oct 30 16:22:38.713664 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.713647 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-82w6j"] Oct 30 16:22:38.714990 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.713855 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:38.718611 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.718595 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mfwcr"] Oct 30 16:22:38.719891 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.718811 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:38.721873 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.721837 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5r74t"] Oct 30 16:22:38.721873 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.721873 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9b9bn"] Oct 30 16:22:38.722059 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.721895 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2sc7w"] Oct 30 16:22:38.722059 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.721914 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rmgmf"] Oct 30 16:22:38.722059 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.721936 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm"] Oct 30 16:22:38.722059 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.721958 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8"] Oct 30 16:22:38.722059 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.721978 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-58c95ddf47-6m2jw"] Oct 30 16:22:38.722059 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.721997 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44"] Oct 30 16:22:38.722059 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.722017 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kx55q"] Oct 30 16:22:38.722059 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.722036 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-szx5s"] Oct 30 16:22:38.722059 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.722055 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mfwcr"] Oct 30 16:22:38.722456 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.722129 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mfwcr" Oct 30 16:22:38.722876 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.722860 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6xx9g"] Oct 30 16:22:38.724813 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.724791 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp"] Oct 30 16:22:38.726052 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.726035 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-sj6h8"] Oct 30 16:22:38.726683 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.726662 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf"] Oct 30 16:22:38.746304 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.746282 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-82w6j"] Oct 30 16:22:38.747146 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.747126 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dfdc4b7f-lkk6g"] Oct 30 16:22:38.780269 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.780247 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 16:22:38.780416 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.780312 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 16:22:38.781254 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.780341 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:38.781254 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.780681 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 16:22:38.781419 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.780705 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:38.781419 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.781267 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:22:38.781419 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.780858 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 16:22:38.781419 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.780920 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:38.781419 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.780987 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 16:22:38.781419 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.781013 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 16:22:38.781419 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.781039 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:38.781419 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.781208 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 16:22:38.781419 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.780714 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:38.782949 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.782921 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 16:22:38.783038 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.782925 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:38.783101 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.783090 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 16:22:38.783161 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.783121 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 16:22:38.783161 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.783125 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:38.783161 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.783146 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 16:22:38.783297 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.783223 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 16:22:38.783297 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.783258 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 16:22:38.794289 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.794271 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 16:22:38.794424 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.794407 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:22:38.794504 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.794441 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 16:22:38.794504 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.794483 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:38.794584 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.794545 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:38.794713 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.794696 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 16:22:38.795050 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.795029 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 16:22:38.796861 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.795242 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:38.796861 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.795673 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 16:22:38.796861 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.796699 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 16:22:38.799440 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799418 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt2sw\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-kube-api-access-zt2sw\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.799516 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799447 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66fd7\" (UniqueName: \"kubernetes.io/projected/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-kube-api-access-66fd7\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:38.799516 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799465 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kghb4\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-kube-api-access-kghb4\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.799516 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799481 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcgvd\" (UniqueName: \"kubernetes.io/projected/4dfa9a2f-f22b-4c85-a594-903928b2034a-kube-api-access-dcgvd\") pod \"ingress-canary-mfwcr\" (UID: \"4dfa9a2f-f22b-4c85-a594-903928b2034a\") " pod="openshift-ingress-canary/ingress-canary-mfwcr" Oct 30 16:22:38.799516 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799508 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc6nb\" (UniqueName: \"kubernetes.io/projected/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-kube-api-access-fc6nb\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:38.799691 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799562 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f79b5c8d-c736-4db0-93b4-b6a40ad58955-tmp-dir\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:38.799691 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799596 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-image-registry-private-configuration\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.799691 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799650 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgr65\" (UniqueName: \"kubernetes.io/projected/9c282411-f8cb-4c83-8959-52035df7cd24-kube-api-access-cgr65\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.799831 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799690 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqhlw\" (UniqueName: \"kubernetes.io/projected/a21d3561-30d9-4a48-816b-5b7be4e5813d-kube-api-access-lqhlw\") pod \"volume-data-source-validator-58c9f5f46d-rmgmf\" (UID: \"a21d3561-30d9-4a48-816b-5b7be4e5813d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rmgmf" Oct 30 16:22:38.799831 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799743 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-bound-sa-token\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.799831 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799802 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2zxc\" (UniqueName: \"kubernetes.io/projected/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-kube-api-access-n2zxc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wgr44\" (UID: \"a1e1a5f5-7d62-43a9-9a51-1cb702e80963\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:38.799992 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799838 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:38.799992 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799855 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t929j\" (UniqueName: \"kubernetes.io/projected/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-kube-api-access-t929j\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:38.799992 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799870 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dfa9a2f-f22b-4c85-a594-903928b2034a-cert\") pod \"ingress-canary-mfwcr\" (UID: \"4dfa9a2f-f22b-4c85-a594-903928b2034a\") " pod="openshift-ingress-canary/ingress-canary-mfwcr" Oct 30 16:22:38.799992 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-default-certificate\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:38.799992 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799921 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-installation-pull-secrets\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.799992 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799958 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-bound-sa-token\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.799992 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.799982 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qglw\" (UniqueName: \"kubernetes.io/projected/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-kube-api-access-4qglw\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:38.800223 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800001 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c282411-f8cb-4c83-8959-52035df7cd24-tmp\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.800223 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800040 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6xx9g\" (UID: \"78ae398a-fdb1-48f8-94fa-51eb582e663a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:38.800223 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9c282411-f8cb-4c83-8959-52035df7cd24-snapshots\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.800223 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800098 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwpb8\" (UniqueName: \"kubernetes.io/projected/1026393f-ba8e-408a-b7df-a1d03e047b1f-kube-api-access-nwpb8\") pod \"kas-connection-checker-685895c7c9-5r74t\" (UID: \"1026393f-ba8e-408a-b7df-a1d03e047b1f\") " pod="kube-system/kas-connection-checker-685895c7c9-5r74t" Oct 30 16:22:38.800223 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800119 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxmwp\" (UniqueName: \"kubernetes.io/projected/f79b5c8d-c736-4db0-93b4-b6a40ad58955-kube-api-access-gxmwp\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:38.800223 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800141 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:38.800223 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800188 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.800223 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-installation-pull-secrets\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.800432 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800246 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wgr44\" (UID: \"a1e1a5f5-7d62-43a9-9a51-1cb702e80963\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:38.800432 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800294 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f79b5c8d-c736-4db0-93b4-b6a40ad58955-config-volume\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:38.800504 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800455 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wgr44\" (UID: \"a1e1a5f5-7d62-43a9-9a51-1cb702e80963\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:38.800534 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800524 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-certificates\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.800576 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800554 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-image-registry-private-configuration\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.800607 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/78ae398a-fdb1-48f8-94fa-51eb582e663a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6xx9g\" (UID: \"78ae398a-fdb1-48f8-94fa-51eb582e663a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:38.800659 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800644 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:38.800691 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800678 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4922dc9-bd15-45a0-a9f6-e61089e6096b-config\") pod \"service-ca-operator-544d749c8c-p79fm\" (UID: \"a4922dc9-bd15-45a0-a9f6-e61089e6096b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:38.800762 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800703 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-stats-auth\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:38.800809 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800794 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:38.800839 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800821 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf7x5\" (UniqueName: \"kubernetes.io/projected/49aa267f-cf95-4d9b-98c4-df106fcd9620-kube-api-access-nf7x5\") pod \"kas-connection-checker-685895c7c9-2sc7w\" (UID: \"49aa267f-cf95-4d9b-98c4-df106fcd9620\") " pod="kube-system/kas-connection-checker-685895c7c9-2sc7w" Oct 30 16:22:38.800883 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800859 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4922dc9-bd15-45a0-a9f6-e61089e6096b-serving-cert\") pod \"service-ca-operator-544d749c8c-p79fm\" (UID: \"a4922dc9-bd15-45a0-a9f6-e61089e6096b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:38.800919 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800905 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c282411-f8cb-4c83-8959-52035df7cd24-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.800950 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800936 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-ca-trust-extracted\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.800980 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800960 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:38.801014 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.800989 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:38.801042 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.801021 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-certificates\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.801086 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.801072 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-trusted-ca\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:38.801148 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.801135 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-trusted-ca\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.801177 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.801162 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-config\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:38.801209 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.801188 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh7x7\" (UniqueName: \"kubernetes.io/projected/c8ddcddb-5053-4950-bc4e-5d8f83ec8f08-kube-api-access-qh7x7\") pod \"kas-connection-checker-685895c7c9-9b9bn\" (UID: \"c8ddcddb-5053-4950-bc4e-5d8f83ec8f08\") " pod="kube-system/kas-connection-checker-685895c7c9-9b9bn" Oct 30 16:22:38.801269 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.801252 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:38.801300 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.801280 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c282411-f8cb-4c83-8959-52035df7cd24-serving-cert\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.801300 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.801296 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7932d7cc-b9fe-423f-bc88-93711eaec908-ca-trust-extracted\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.801368 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.801327 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.801368 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.801343 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-trusted-ca\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.801368 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.801363 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldgcm\" (UniqueName: \"kubernetes.io/projected/a4922dc9-bd15-45a0-a9f6-e61089e6096b-kube-api-access-ldgcm\") pod \"service-ca-operator-544d749c8c-p79fm\" (UID: \"a4922dc9-bd15-45a0-a9f6-e61089e6096b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:38.801512 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.801376 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q6vt\" (UniqueName: \"kubernetes.io/projected/12874a88-98a6-436e-973d-cb931fc769de-kube-api-access-8q6vt\") pod \"network-check-source-6f78d59f99-sj6h8\" (UID: \"12874a88-98a6-436e-973d-cb931fc769de\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sj6h8" Oct 30 16:22:38.801512 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.801426 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c282411-f8cb-4c83-8959-52035df7cd24-service-ca-bundle\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.801512 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.801449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-serving-cert\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:38.808300 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808135 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 16:22:38.808300 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808140 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 16:22:38.808300 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808195 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 16:22:38.808300 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808207 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 16:22:38.808300 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808270 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 16:22:38.808594 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808334 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-65hn8\"" Oct 30 16:22:38.808594 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808514 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 16:22:38.808594 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808574 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 16:22:38.808752 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808598 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bd59p\"" Oct 30 16:22:38.808752 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808669 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 16:22:38.808752 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808680 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 16:22:38.808752 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808681 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cn6jt\"" Oct 30 16:22:38.808752 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808714 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 16:22:38.808752 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808686 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 16:22:38.809210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808757 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 16:22:38.809210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808764 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z9zpr\"" Oct 30 16:22:38.809210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808770 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-vqbh2\"" Oct 30 16:22:38.809210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808824 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6n6gp\"" Oct 30 16:22:38.809210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808834 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wm4jw\"" Oct 30 16:22:38.809210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808865 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 16:22:38.809210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808891 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-nnwcz\"" Oct 30 16:22:38.809210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808906 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lwtv6\"" Oct 30 16:22:38.809210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808906 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xdq5p\"" Oct 30 16:22:38.809210 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.808970 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wkbtq\"" Oct 30 16:22:38.901827 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.901780 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-bound-sa-token\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.901827 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.901803 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qglw\" (UniqueName: \"kubernetes.io/projected/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-kube-api-access-4qglw\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:38.901827 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.901819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c282411-f8cb-4c83-8959-52035df7cd24-tmp\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.902033 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.901836 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6xx9g\" (UID: \"78ae398a-fdb1-48f8-94fa-51eb582e663a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:38.902033 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.901859 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9c282411-f8cb-4c83-8959-52035df7cd24-snapshots\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.902033 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.901872 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwpb8\" (UniqueName: \"kubernetes.io/projected/1026393f-ba8e-408a-b7df-a1d03e047b1f-kube-api-access-nwpb8\") pod \"kas-connection-checker-685895c7c9-5r74t\" (UID: \"1026393f-ba8e-408a-b7df-a1d03e047b1f\") " pod="kube-system/kas-connection-checker-685895c7c9-5r74t" Oct 30 16:22:38.902033 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.901888 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxmwp\" (UniqueName: \"kubernetes.io/projected/f79b5c8d-c736-4db0-93b4-b6a40ad58955-kube-api-access-gxmwp\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:38.902033 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.901904 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:38.902033 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.901982 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:22:38.902316 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.902039 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert podName:78ae398a-fdb1-48f8-94fa-51eb582e663a nodeName:}" failed. No retries permitted until 2026-10-30 16:22:39.402022186 +0000 UTC m=+30.636645832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6xx9g" (UID: "78ae398a-fdb1-48f8-94fa-51eb582e663a") : secret "networking-console-plugin-cert" not found Oct 30 16:22:38.902316 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.902316 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902134 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-installation-pull-secrets\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.902316 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902168 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wgr44\" (UID: \"a1e1a5f5-7d62-43a9-9a51-1cb702e80963\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:38.902316 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902188 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c282411-f8cb-4c83-8959-52035df7cd24-tmp\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.902316 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902196 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f79b5c8d-c736-4db0-93b4-b6a40ad58955-config-volume\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:38.902316 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.902178 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:22:38.902316 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.902214 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dfdc4b7f-lkk6g: secret "image-registry-tls" not found Oct 30 16:22:38.902316 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wgr44\" (UID: \"a1e1a5f5-7d62-43a9-9a51-1cb702e80963\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:38.902316 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.902254 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls podName:7932d7cc-b9fe-423f-bc88-93711eaec908 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:39.402240745 +0000 UTC m=+30.636864400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls") pod "image-registry-5dfdc4b7f-lkk6g" (UID: "7932d7cc-b9fe-423f-bc88-93711eaec908") : secret "image-registry-tls" not found Oct 30 16:22:38.902316 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902274 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-certificates\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.902316 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902307 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-image-registry-private-configuration\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.902892 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902333 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/78ae398a-fdb1-48f8-94fa-51eb582e663a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6xx9g\" (UID: \"78ae398a-fdb1-48f8-94fa-51eb582e663a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:38.902892 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902359 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:38.902892 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902383 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4922dc9-bd15-45a0-a9f6-e61089e6096b-config\") pod \"service-ca-operator-544d749c8c-p79fm\" (UID: \"a4922dc9-bd15-45a0-a9f6-e61089e6096b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:38.902892 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-stats-auth\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:38.902892 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902427 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:38.902892 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.902502 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:22:38.902892 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.902573 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs podName:1ab95e94-bdff-4f1f-a0fe-095a2509d6f8 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:39.40255798 +0000 UTC m=+30.637181622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs") pod "router-default-58c95ddf47-6m2jw" (UID: "1ab95e94-bdff-4f1f-a0fe-095a2509d6f8") : secret "router-metrics-certs-default" not found Oct 30 16:22:38.902892 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902574 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9c282411-f8cb-4c83-8959-52035df7cd24-snapshots\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.902892 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902615 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf7x5\" (UniqueName: \"kubernetes.io/projected/49aa267f-cf95-4d9b-98c4-df106fcd9620-kube-api-access-nf7x5\") pod \"kas-connection-checker-685895c7c9-2sc7w\" (UID: \"49aa267f-cf95-4d9b-98c4-df106fcd9620\") " pod="kube-system/kas-connection-checker-685895c7c9-2sc7w" Oct 30 16:22:38.902892 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902643 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4922dc9-bd15-45a0-a9f6-e61089e6096b-serving-cert\") pod \"service-ca-operator-544d749c8c-p79fm\" (UID: \"a4922dc9-bd15-45a0-a9f6-e61089e6096b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:38.902892 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902678 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c282411-f8cb-4c83-8959-52035df7cd24-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.902892 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902704 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-ca-trust-extracted\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.902892 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902756 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:38.902892 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902762 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:38.902892 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902791 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:38.902892 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-certificates\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.903553 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902850 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-trusted-ca\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:38.903553 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902880 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-trusted-ca\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.903553 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902897 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wgr44\" (UID: \"a1e1a5f5-7d62-43a9-9a51-1cb702e80963\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:38.903553 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902923 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f79b5c8d-c736-4db0-93b4-b6a40ad58955-config-volume\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:38.903553 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902947 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-certificates\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.903553 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.903117 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4922dc9-bd15-45a0-a9f6-e61089e6096b-config\") pod \"service-ca-operator-544d749c8c-p79fm\" (UID: \"a4922dc9-bd15-45a0-a9f6-e61089e6096b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:38.903553 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.902509 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:22:38.903553 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.903183 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls podName:7fff2d07-9f9e-4edb-a32c-ad3a9440fc17 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:39.403169379 +0000 UTC m=+30.637793035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-btqzf" (UID: "7fff2d07-9f9e-4edb-a32c-ad3a9440fc17") : secret "samples-operator-tls" not found Oct 30 16:22:38.903553 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.903193 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:22:38.903553 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.903241 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls podName:f79b5c8d-c736-4db0-93b4-b6a40ad58955 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:39.403226048 +0000 UTC m=+30.637849693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls") pod "dns-default-82w6j" (UID: "f79b5c8d-c736-4db0-93b4-b6a40ad58955") : secret "dns-default-metrics-tls" not found Oct 30 16:22:38.903971 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.903586 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:38.903971 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.903642 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/78ae398a-fdb1-48f8-94fa-51eb582e663a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6xx9g\" (UID: \"78ae398a-fdb1-48f8-94fa-51eb582e663a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:38.903971 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.902903 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-config\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:38.903971 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.903694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh7x7\" (UniqueName: \"kubernetes.io/projected/c8ddcddb-5053-4950-bc4e-5d8f83ec8f08-kube-api-access-qh7x7\") pod \"kas-connection-checker-685895c7c9-9b9bn\" (UID: \"c8ddcddb-5053-4950-bc4e-5d8f83ec8f08\") " pod="kube-system/kas-connection-checker-685895c7c9-9b9bn" Oct 30 16:22:38.903971 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.903757 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:38.903971 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.903785 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c282411-f8cb-4c83-8959-52035df7cd24-serving-cert\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.903971 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.903807 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7932d7cc-b9fe-423f-bc88-93711eaec908-ca-trust-extracted\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.903971 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.903835 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.903971 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.903839 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-certificates\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.903971 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.903847 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-ca-trust-extracted\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.903971 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.903865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-trusted-ca\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.903971 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.903894 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldgcm\" (UniqueName: \"kubernetes.io/projected/a4922dc9-bd15-45a0-a9f6-e61089e6096b-kube-api-access-ldgcm\") pod \"service-ca-operator-544d749c8c-p79fm\" (UID: \"a4922dc9-bd15-45a0-a9f6-e61089e6096b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:38.903971 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.903921 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q6vt\" (UniqueName: \"kubernetes.io/projected/12874a88-98a6-436e-973d-cb931fc769de-kube-api-access-8q6vt\") pod \"network-check-source-6f78d59f99-sj6h8\" (UID: \"12874a88-98a6-436e-973d-cb931fc769de\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sj6h8" Oct 30 16:22:38.903971 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.903952 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c282411-f8cb-4c83-8959-52035df7cd24-service-ca-bundle\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.904755 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.903981 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-serving-cert\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:38.904755 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904010 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt2sw\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-kube-api-access-zt2sw\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.904755 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904035 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66fd7\" (UniqueName: \"kubernetes.io/projected/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-kube-api-access-66fd7\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:38.904755 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904061 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kghb4\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-kube-api-access-kghb4\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.904755 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904084 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c282411-f8cb-4c83-8959-52035df7cd24-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.904755 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904087 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcgvd\" (UniqueName: \"kubernetes.io/projected/4dfa9a2f-f22b-4c85-a594-903928b2034a-kube-api-access-dcgvd\") pod \"ingress-canary-mfwcr\" (UID: \"4dfa9a2f-f22b-4c85-a594-903928b2034a\") " pod="openshift-ingress-canary/ingress-canary-mfwcr" Oct 30 16:22:38.904755 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904114 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc6nb\" (UniqueName: \"kubernetes.io/projected/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-kube-api-access-fc6nb\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:38.904755 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f79b5c8d-c736-4db0-93b4-b6a40ad58955-tmp-dir\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:38.904755 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-image-registry-private-configuration\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.904755 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904198 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgr65\" (UniqueName: \"kubernetes.io/projected/9c282411-f8cb-4c83-8959-52035df7cd24-kube-api-access-cgr65\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.904755 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904226 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqhlw\" (UniqueName: \"kubernetes.io/projected/a21d3561-30d9-4a48-816b-5b7be4e5813d-kube-api-access-lqhlw\") pod \"volume-data-source-validator-58c9f5f46d-rmgmf\" (UID: \"a21d3561-30d9-4a48-816b-5b7be4e5813d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rmgmf" Oct 30 16:22:38.904755 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904251 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-bound-sa-token\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.904755 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904281 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2zxc\" (UniqueName: \"kubernetes.io/projected/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-kube-api-access-n2zxc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wgr44\" (UID: \"a1e1a5f5-7d62-43a9-9a51-1cb702e80963\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:38.904755 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:38.904755 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904344 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t929j\" (UniqueName: \"kubernetes.io/projected/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-kube-api-access-t929j\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:38.904755 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904367 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dfa9a2f-f22b-4c85-a594-903928b2034a-cert\") pod \"ingress-canary-mfwcr\" (UID: \"4dfa9a2f-f22b-4c85-a594-903928b2034a\") " pod="openshift-ingress-canary/ingress-canary-mfwcr" Oct 30 16:22:38.905503 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904399 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-default-certificate\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:38.905503 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.904445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-installation-pull-secrets\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.906153 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.905840 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f79b5c8d-c736-4db0-93b4-b6a40ad58955-tmp-dir\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:38.906153 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.905997 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-config\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:38.906153 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.906112 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-trusted-ca\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:38.906356 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.906298 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-trusted-ca\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.906758 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.906481 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7932d7cc-b9fe-423f-bc88-93711eaec908-ca-trust-extracted\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.906758 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.906581 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:22:38.906758 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.906601 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8: secret "image-registry-tls" not found Oct 30 16:22:38.906758 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.906676 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:38.906758 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.906711 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls podName:fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:39.406681969 +0000 UTC m=+30.641305610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls") pod "image-registry-5cc8d7fd6b-v2gt8" (UID: "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5") : secret "image-registry-tls" not found Oct 30 16:22:38.908485 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.907632 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle podName:1ab95e94-bdff-4f1f-a0fe-095a2509d6f8 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:39.407612971 +0000 UTC m=+30.642236603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle") pod "router-default-58c95ddf47-6m2jw" (UID: "1ab95e94-bdff-4f1f-a0fe-095a2509d6f8") : configmap references non-existent config key: service-ca.crt Oct 30 16:22:38.908485 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:38.907657 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls podName:93b8a9b8-43df-4dc0-97da-4bfd11f373cd nodeName:}" failed. No retries permitted until 2026-10-30 16:22:39.40764536 +0000 UTC m=+30.642268988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gn2hp" (UID: "93b8a9b8-43df-4dc0-97da-4bfd11f373cd") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:38.909967 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.909505 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-trusted-ca\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.910437 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.910414 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-default-certificate\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:38.910543 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.910453 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wgr44\" (UID: \"a1e1a5f5-7d62-43a9-9a51-1cb702e80963\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:38.910543 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.910476 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4922dc9-bd15-45a0-a9f6-e61089e6096b-serving-cert\") pod \"service-ca-operator-544d749c8c-p79fm\" (UID: \"a4922dc9-bd15-45a0-a9f6-e61089e6096b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:38.910543 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.910536 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-stats-auth\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:38.910963 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.910943 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c282411-f8cb-4c83-8959-52035df7cd24-service-ca-bundle\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.911030 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.910993 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c282411-f8cb-4c83-8959-52035df7cd24-serving-cert\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.912596 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.912568 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-image-registry-private-configuration\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.915656 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.915630 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-installation-pull-secrets\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.915859 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.915754 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-image-registry-private-configuration\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.916800 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.916706 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-installation-pull-secrets\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.916925 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.916765 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dfa9a2f-f22b-4c85-a594-903928b2034a-cert\") pod \"ingress-canary-mfwcr\" (UID: \"4dfa9a2f-f22b-4c85-a594-903928b2034a\") " pod="openshift-ingress-canary/ingress-canary-mfwcr" Oct 30 16:22:38.916925 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.916848 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:38.917164 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.917145 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-serving-cert\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:38.932404 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.932378 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc6nb\" (UniqueName: \"kubernetes.io/projected/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-kube-api-access-fc6nb\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:38.932601 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.932583 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxmwp\" (UniqueName: \"kubernetes.io/projected/f79b5c8d-c736-4db0-93b4-b6a40ad58955-kube-api-access-gxmwp\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:38.932661 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.932642 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf7x5\" (UniqueName: \"kubernetes.io/projected/49aa267f-cf95-4d9b-98c4-df106fcd9620-kube-api-access-nf7x5\") pod \"kas-connection-checker-685895c7c9-2sc7w\" (UID: \"49aa267f-cf95-4d9b-98c4-df106fcd9620\") " pod="kube-system/kas-connection-checker-685895c7c9-2sc7w" Oct 30 16:22:38.933749 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.933702 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwpb8\" (UniqueName: \"kubernetes.io/projected/1026393f-ba8e-408a-b7df-a1d03e047b1f-kube-api-access-nwpb8\") pod \"kas-connection-checker-685895c7c9-5r74t\" (UID: \"1026393f-ba8e-408a-b7df-a1d03e047b1f\") " pod="kube-system/kas-connection-checker-685895c7c9-5r74t" Oct 30 16:22:38.934207 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.934187 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qglw\" (UniqueName: \"kubernetes.io/projected/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-kube-api-access-4qglw\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:38.935241 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.935219 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-bound-sa-token\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.965082 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.965062 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgr65\" (UniqueName: \"kubernetes.io/projected/9c282411-f8cb-4c83-8959-52035df7cd24-kube-api-access-cgr65\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:38.965514 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.965487 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh7x7\" (UniqueName: \"kubernetes.io/projected/c8ddcddb-5053-4950-bc4e-5d8f83ec8f08-kube-api-access-qh7x7\") pod \"kas-connection-checker-685895c7c9-9b9bn\" (UID: \"c8ddcddb-5053-4950-bc4e-5d8f83ec8f08\") " pod="kube-system/kas-connection-checker-685895c7c9-9b9bn" Oct 30 16:22:38.965809 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.965793 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2zxc\" (UniqueName: \"kubernetes.io/projected/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-kube-api-access-n2zxc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wgr44\" (UID: \"a1e1a5f5-7d62-43a9-9a51-1cb702e80963\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:38.966568 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.966548 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q6vt\" (UniqueName: \"kubernetes.io/projected/12874a88-98a6-436e-973d-cb931fc769de-kube-api-access-8q6vt\") pod \"network-check-source-6f78d59f99-sj6h8\" (UID: \"12874a88-98a6-436e-973d-cb931fc769de\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sj6h8" Oct 30 16:22:38.966662 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.966644 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66fd7\" (UniqueName: \"kubernetes.io/projected/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-kube-api-access-66fd7\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:38.968952 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.968929 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kghb4\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-kube-api-access-kghb4\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.969089 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.969072 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-bound-sa-token\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:38.972460 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.972439 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t929j\" (UniqueName: \"kubernetes.io/projected/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-kube-api-access-t929j\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:38.973646 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.973623 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcgvd\" (UniqueName: \"kubernetes.io/projected/4dfa9a2f-f22b-4c85-a594-903928b2034a-kube-api-access-dcgvd\") pod \"ingress-canary-mfwcr\" (UID: \"4dfa9a2f-f22b-4c85-a594-903928b2034a\") " pod="openshift-ingress-canary/ingress-canary-mfwcr" Oct 30 16:22:38.973919 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.973898 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5r74t" Oct 30 16:22:38.974125 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.974106 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldgcm\" (UniqueName: \"kubernetes.io/projected/a4922dc9-bd15-45a0-a9f6-e61089e6096b-kube-api-access-ldgcm\") pod \"service-ca-operator-544d749c8c-p79fm\" (UID: \"a4922dc9-bd15-45a0-a9f6-e61089e6096b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:38.974505 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.974485 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt2sw\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-kube-api-access-zt2sw\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:38.974642 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.974628 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqhlw\" (UniqueName: \"kubernetes.io/projected/a21d3561-30d9-4a48-816b-5b7be4e5813d-kube-api-access-lqhlw\") pod \"volume-data-source-validator-58c9f5f46d-rmgmf\" (UID: \"a21d3561-30d9-4a48-816b-5b7be4e5813d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rmgmf" Oct 30 16:22:38.981804 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.981704 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9b9bn" Oct 30 16:22:38.991642 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:38.991620 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-2sc7w" Oct 30 16:22:39.031368 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.031197 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rmgmf" Oct 30 16:22:39.038483 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.038289 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:39.053976 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.053740 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:39.060791 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.059489 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:39.067682 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.067414 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:39.079439 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.079417 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sj6h8" Oct 30 16:22:39.099463 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.098539 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mfwcr" Oct 30 16:22:39.142574 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.138694 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:39.283436 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.283406 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:39.305337 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.305299 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9b9bn"] Oct 30 16:22:39.308818 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.308776 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-2sc7w"] Oct 30 16:22:39.309299 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.309280 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5r74t"] Oct 30 16:22:39.313698 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:39.313677 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49aa267f_cf95_4d9b_98c4_df106fcd9620.slice/crio-aafa2232431d7aa08a6ec916db80c0e6faef13bb2d75d93016de45d34f48c0db WatchSource:0}: Error finding container aafa2232431d7aa08a6ec916db80c0e6faef13bb2d75d93016de45d34f48c0db: Status 404 returned error can't find the container with id aafa2232431d7aa08a6ec916db80c0e6faef13bb2d75d93016de45d34f48c0db Oct 30 16:22:39.318189 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:39.318169 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1026393f_ba8e_408a_b7df_a1d03e047b1f.slice/crio-b8df4a6db68add4fac3ed4114f9358268d5b346bf3b985632487b098d33f2034 WatchSource:0}: Error finding container b8df4a6db68add4fac3ed4114f9358268d5b346bf3b985632487b098d33f2034: Status 404 returned error can't find the container with id b8df4a6db68add4fac3ed4114f9358268d5b346bf3b985632487b098d33f2034 Oct 30 16:22:39.380200 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.380170 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rmgmf"] Oct 30 16:22:39.393915 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:39.393883 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21d3561_30d9_4a48_816b_5b7be4e5813d.slice/crio-065bc8e2e2cc3204f157f053f695f5c1b35a88bd84faa84ae93555c2a70cbf9a WatchSource:0}: Error finding container 065bc8e2e2cc3204f157f053f695f5c1b35a88bd84faa84ae93555c2a70cbf9a: Status 404 returned error can't find the container with id 065bc8e2e2cc3204f157f053f695f5c1b35a88bd84faa84ae93555c2a70cbf9a Oct 30 16:22:39.418151 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.418126 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:39.418236 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.418169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:39.418236 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.418187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:39.418236 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.418211 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:39.418375 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418280 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:22:39.418375 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418286 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:22:39.418375 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418283 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:22:39.418375 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418329 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dfdc4b7f-lkk6g: secret "image-registry-tls" not found Oct 30 16:22:39.418375 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.418343 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:39.418375 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418351 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:22:39.418375 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418364 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls podName:f79b5c8d-c736-4db0-93b4-b6a40ad58955 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:40.418345902 +0000 UTC m=+31.652969536 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls") pod "dns-default-82w6j" (UID: "f79b5c8d-c736-4db0-93b4-b6a40ad58955") : secret "dns-default-metrics-tls" not found Oct 30 16:22:39.418603 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418392 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs podName:1ab95e94-bdff-4f1f-a0fe-095a2509d6f8 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:40.418383926 +0000 UTC m=+31.653007553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs") pod "router-default-58c95ddf47-6m2jw" (UID: "1ab95e94-bdff-4f1f-a0fe-095a2509d6f8") : secret "router-metrics-certs-default" not found Oct 30 16:22:39.418603 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418399 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:39.418603 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418423 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls podName:7932d7cc-b9fe-423f-bc88-93711eaec908 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:40.418415152 +0000 UTC m=+31.653038787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls") pod "image-registry-5dfdc4b7f-lkk6g" (UID: "7932d7cc-b9fe-423f-bc88-93711eaec908") : secret "image-registry-tls" not found Oct 30 16:22:39.418603 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418441 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls podName:7fff2d07-9f9e-4edb-a32c-ad3a9440fc17 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:40.418433525 +0000 UTC m=+31.653057201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-btqzf" (UID: "7fff2d07-9f9e-4edb-a32c-ad3a9440fc17") : secret "samples-operator-tls" not found Oct 30 16:22:39.418603 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418458 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls podName:93b8a9b8-43df-4dc0-97da-4bfd11f373cd nodeName:}" failed. No retries permitted until 2026-10-30 16:22:40.418450059 +0000 UTC m=+31.653073719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gn2hp" (UID: "93b8a9b8-43df-4dc0-97da-4bfd11f373cd") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:39.418603 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.418499 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:39.418603 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.418560 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:39.418603 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418591 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:22:39.418603 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418601 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8: secret "image-registry-tls" not found Oct 30 16:22:39.418944 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.418624 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6xx9g\" (UID: \"78ae398a-fdb1-48f8-94fa-51eb582e663a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:39.418944 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418636 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls podName:fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:40.418626236 +0000 UTC m=+31.653249883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls") pod "image-registry-5cc8d7fd6b-v2gt8" (UID: "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5") : secret "image-registry-tls" not found Oct 30 16:22:39.418944 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418687 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle podName:1ab95e94-bdff-4f1f-a0fe-095a2509d6f8 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:40.418675609 +0000 UTC m=+31.653299251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle") pod "router-default-58c95ddf47-6m2jw" (UID: "1ab95e94-bdff-4f1f-a0fe-095a2509d6f8") : configmap references non-existent config key: service-ca.crt Oct 30 16:22:39.418944 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418761 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:22:39.418944 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:39.418799 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert podName:78ae398a-fdb1-48f8-94fa-51eb582e663a nodeName:}" failed. No retries permitted until 2026-10-30 16:22:40.418792001 +0000 UTC m=+31.653415628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6xx9g" (UID: "78ae398a-fdb1-48f8-94fa-51eb582e663a") : secret "networking-console-plugin-cert" not found Oct 30 16:22:39.440155 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.439755 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm"] Oct 30 16:22:39.457280 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.454863 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44"] Oct 30 16:22:39.457280 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.455658 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kx55q"] Oct 30 16:22:39.457445 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.457415 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-szx5s"] Oct 30 16:22:39.491622 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.491598 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-sj6h8"] Oct 30 16:22:39.495029 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:39.495003 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12874a88_98a6_436e_973d_cb931fc769de.slice/crio-6ceea6bbf77a56ab19647fffef40a9c6dfec70a770767769ce9b22bf5d03d0a2 WatchSource:0}: Error finding container 6ceea6bbf77a56ab19647fffef40a9c6dfec70a770767769ce9b22bf5d03d0a2: Status 404 returned error can't find the container with id 6ceea6bbf77a56ab19647fffef40a9c6dfec70a770767769ce9b22bf5d03d0a2 Oct 30 16:22:39.496050 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.496028 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mfwcr"] Oct 30 16:22:39.498684 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:39.498660 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfa9a2f_f22b_4c85_a594_903928b2034a.slice/crio-f5967c9990da73ecc128bfd5f29740b925371d8923b2b5015f791f1d88ef1905 WatchSource:0}: Error finding container f5967c9990da73ecc128bfd5f29740b925371d8923b2b5015f791f1d88ef1905: Status 404 returned error can't find the container with id f5967c9990da73ecc128bfd5f29740b925371d8923b2b5015f791f1d88ef1905 Oct 30 16:22:39.612175 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.612151 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mfwcr" event={"ID":"4dfa9a2f-f22b-4c85-a594-903928b2034a","Type":"ContainerStarted","Data":"f5967c9990da73ecc128bfd5f29740b925371d8923b2b5015f791f1d88ef1905"} Oct 30 16:22:39.613376 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.613354 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kx55q" event={"ID":"9c282411-f8cb-4c83-8959-52035df7cd24","Type":"ContainerStarted","Data":"7b233a8a715e2f102d61d0b92dedf99637994546b6ecf815e5702e562c961ce6"} Oct 30 16:22:39.615019 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.614997 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9b9bn" event={"ID":"c8ddcddb-5053-4950-bc4e-5d8f83ec8f08","Type":"ContainerStarted","Data":"57463ce06f7470203df3fee6e6a92db3ea8b1f2650747f7d702936fa9618033b"} Oct 30 16:22:39.615116 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.615038 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9b9bn" event={"ID":"c8ddcddb-5053-4950-bc4e-5d8f83ec8f08","Type":"ContainerStarted","Data":"d9fd7b99fb365a96dc6069ed563255afadfd76487c90447756c14152518ff8bb"} Oct 30 16:22:39.616201 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.616178 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" event={"ID":"a1e1a5f5-7d62-43a9-9a51-1cb702e80963","Type":"ContainerStarted","Data":"36e6fc788aeb4842a93d5bf9647b4568b11f62808fd1e38e44f7afac45ef82f8"} Oct 30 16:22:39.617382 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.617363 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rmgmf" event={"ID":"a21d3561-30d9-4a48-816b-5b7be4e5813d","Type":"ContainerStarted","Data":"065bc8e2e2cc3204f157f053f695f5c1b35a88bd84faa84ae93555c2a70cbf9a"} Oct 30 16:22:39.618828 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.618805 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5r74t" event={"ID":"1026393f-ba8e-408a-b7df-a1d03e047b1f","Type":"ContainerStarted","Data":"0e5afafcecb2ba968944422f47a2a72ca4e3cacc7985c3cccca876ae2afc33de"} Oct 30 16:22:39.618921 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.618850 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5r74t" event={"ID":"1026393f-ba8e-408a-b7df-a1d03e047b1f","Type":"ContainerStarted","Data":"b8df4a6db68add4fac3ed4114f9358268d5b346bf3b985632487b098d33f2034"} Oct 30 16:22:39.620032 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.620011 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sj6h8" event={"ID":"12874a88-98a6-436e-973d-cb931fc769de","Type":"ContainerStarted","Data":"6ceea6bbf77a56ab19647fffef40a9c6dfec70a770767769ce9b22bf5d03d0a2"} Oct 30 16:22:39.621051 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.621032 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" event={"ID":"5a25e3b8-4f22-4fdb-86a9-70d1c594236d","Type":"ContainerStarted","Data":"303048ebdd7310eda3df6c6f626aa2573535edeb330043ab110100efe677f7cd"} Oct 30 16:22:39.622200 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.622176 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" event={"ID":"a4922dc9-bd15-45a0-a9f6-e61089e6096b","Type":"ContainerStarted","Data":"a1174aee1f9265a7f668bc519d01c5a5f4cfe4965464c6b18b06f69a16219da3"} Oct 30 16:22:39.623644 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.623624 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2sc7w" event={"ID":"49aa267f-cf95-4d9b-98c4-df106fcd9620","Type":"ContainerStarted","Data":"1daa66136e6eb4e45738ca03d39f9761589ec24837f44caffc139a4de500e0e9"} Oct 30 16:22:39.623735 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.623662 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-2sc7w" event={"ID":"49aa267f-cf95-4d9b-98c4-df106fcd9620","Type":"ContainerStarted","Data":"aafa2232431d7aa08a6ec916db80c0e6faef13bb2d75d93016de45d34f48c0db"} Oct 30 16:22:39.657640 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.657605 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-9b9bn" podStartSLOduration=225.657595204 podStartE2EDuration="3m45.657595204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:39.656762867 +0000 UTC m=+30.891386533" watchObservedRunningTime="2026-10-30 16:22:39.657595204 +0000 UTC m=+30.892218901" Oct 30 16:22:39.700018 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.699938 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5r74t" podStartSLOduration=225.699923072 podStartE2EDuration="3m45.699923072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:39.699454469 +0000 UTC m=+30.934078116" watchObservedRunningTime="2026-10-30 16:22:39.699923072 +0000 UTC m=+30.934546717" Oct 30 16:22:39.743447 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:39.743412 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-2sc7w" podStartSLOduration=225.743400876 podStartE2EDuration="3m45.743400876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:39.743145285 +0000 UTC m=+30.977768934" watchObservedRunningTime="2026-10-30 16:22:39.743400876 +0000 UTC m=+30.978024525" Oct 30 16:22:40.323088 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.321021 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:40.323088 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.321704 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:40.323088 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.322119 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:40.338279 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.338247 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9txpl\"" Oct 30 16:22:40.338588 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.338567 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:22:40.338752 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.338660 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hpqxb\"" Oct 30 16:22:40.338752 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.338737 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:22:40.338867 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.338781 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vhqb4\"" Oct 30 16:22:40.428081 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.428053 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:40.428505 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.428107 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6xx9g\" (UID: \"78ae398a-fdb1-48f8-94fa-51eb582e663a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:40.428505 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.428152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:40.428505 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.428182 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:40.428505 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.428201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:40.428505 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.428233 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:40.428505 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.428288 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:40.428505 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.428321 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:40.428505 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.428433 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:22:40.428505 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.428444 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8: secret "image-registry-tls" not found Oct 30 16:22:40.428505 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.428487 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls podName:fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:42.428475478 +0000 UTC m=+33.663099120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls") pod "image-registry-5cc8d7fd6b-v2gt8" (UID: "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5") : secret "image-registry-tls" not found Oct 30 16:22:40.429362 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.428846 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:22:40.429362 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.428862 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:22:40.429362 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.428905 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs podName:1ab95e94-bdff-4f1f-a0fe-095a2509d6f8 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:42.428888166 +0000 UTC m=+33.663511808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs") pod "router-default-58c95ddf47-6m2jw" (UID: "1ab95e94-bdff-4f1f-a0fe-095a2509d6f8") : secret "router-metrics-certs-default" not found Oct 30 16:22:40.429362 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.428920 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls podName:7fff2d07-9f9e-4edb-a32c-ad3a9440fc17 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:42.428914007 +0000 UTC m=+33.663537635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-btqzf" (UID: "7fff2d07-9f9e-4edb-a32c-ad3a9440fc17") : secret "samples-operator-tls" not found Oct 30 16:22:40.429362 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.428957 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:22:40.429362 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.428966 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle podName:1ab95e94-bdff-4f1f-a0fe-095a2509d6f8 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:42.428960654 +0000 UTC m=+33.663584297 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle") pod "router-default-58c95ddf47-6m2jw" (UID: "1ab95e94-bdff-4f1f-a0fe-095a2509d6f8") : configmap references non-existent config key: service-ca.crt Oct 30 16:22:40.429362 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.428987 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls podName:f79b5c8d-c736-4db0-93b4-b6a40ad58955 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:42.428977283 +0000 UTC m=+33.663600928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls") pod "dns-default-82w6j" (UID: "f79b5c8d-c736-4db0-93b4-b6a40ad58955") : secret "dns-default-metrics-tls" not found Oct 30 16:22:40.429362 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.429017 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:22:40.429362 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.429032 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:40.429362 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.429051 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert podName:78ae398a-fdb1-48f8-94fa-51eb582e663a nodeName:}" failed. No retries permitted until 2026-10-30 16:22:42.42904155 +0000 UTC m=+33.663665210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6xx9g" (UID: "78ae398a-fdb1-48f8-94fa-51eb582e663a") : secret "networking-console-plugin-cert" not found Oct 30 16:22:40.429362 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.429067 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls podName:93b8a9b8-43df-4dc0-97da-4bfd11f373cd nodeName:}" failed. No retries permitted until 2026-10-30 16:22:42.429059119 +0000 UTC m=+33.663682752 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gn2hp" (UID: "93b8a9b8-43df-4dc0-97da-4bfd11f373cd") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:40.429362 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.429092 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:22:40.429362 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.429099 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dfdc4b7f-lkk6g: secret "image-registry-tls" not found Oct 30 16:22:40.429362 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:40.429139 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls podName:7932d7cc-b9fe-423f-bc88-93711eaec908 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:42.429126737 +0000 UTC m=+33.663750392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls") pod "image-registry-5dfdc4b7f-lkk6g" (UID: "7932d7cc-b9fe-423f-bc88-93711eaec908") : secret "image-registry-tls" not found Oct 30 16:22:40.764794 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.764690 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:40.906556 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:40.906515 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:42.355112 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:42.355033 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:42.452330 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:42.452293 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:42.452499 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:42.452366 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:42.452499 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:42.452406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:42.452499 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:42.452460 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:42.452662 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:42.452509 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6xx9g\" (UID: \"78ae398a-fdb1-48f8-94fa-51eb582e663a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:42.452662 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:42.452549 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:42.452662 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:42.452592 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:42.452662 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:42.452619 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:42.452894 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.452735 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:22:42.452894 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.452792 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls podName:7fff2d07-9f9e-4edb-a32c-ad3a9440fc17 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.452779595 +0000 UTC m=+37.687403224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-btqzf" (UID: "7fff2d07-9f9e-4edb-a32c-ad3a9440fc17") : secret "samples-operator-tls" not found Oct 30 16:22:42.453111 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.453096 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:22:42.453148 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.453133 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls podName:f79b5c8d-c736-4db0-93b4-b6a40ad58955 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.453123642 +0000 UTC m=+37.687747269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls") pod "dns-default-82w6j" (UID: "f79b5c8d-c736-4db0-93b4-b6a40ad58955") : secret "dns-default-metrics-tls" not found Oct 30 16:22:42.453184 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.453167 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:42.453215 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.453187 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls podName:93b8a9b8-43df-4dc0-97da-4bfd11f373cd nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.453180773 +0000 UTC m=+37.687804401 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gn2hp" (UID: "93b8a9b8-43df-4dc0-97da-4bfd11f373cd") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:42.453254 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.453223 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:22:42.453254 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.453231 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8: secret "image-registry-tls" not found Oct 30 16:22:42.453254 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.453249 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls podName:fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.453243685 +0000 UTC m=+37.687867313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls") pod "image-registry-5cc8d7fd6b-v2gt8" (UID: "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5") : secret "image-registry-tls" not found Oct 30 16:22:42.453344 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.453287 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle podName:1ab95e94-bdff-4f1f-a0fe-095a2509d6f8 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.453282144 +0000 UTC m=+37.687905772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle") pod "router-default-58c95ddf47-6m2jw" (UID: "1ab95e94-bdff-4f1f-a0fe-095a2509d6f8") : configmap references non-existent config key: service-ca.crt Oct 30 16:22:42.453344 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.453319 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:22:42.453344 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.453337 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert podName:78ae398a-fdb1-48f8-94fa-51eb582e663a nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.453330742 +0000 UTC m=+37.687954370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6xx9g" (UID: "78ae398a-fdb1-48f8-94fa-51eb582e663a") : secret "networking-console-plugin-cert" not found Oct 30 16:22:42.453508 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.453368 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:22:42.453508 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.453374 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dfdc4b7f-lkk6g: secret "image-registry-tls" not found Oct 30 16:22:42.453508 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.453390 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls podName:7932d7cc-b9fe-423f-bc88-93711eaec908 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.45338456 +0000 UTC m=+37.688008188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls") pod "image-registry-5dfdc4b7f-lkk6g" (UID: "7932d7cc-b9fe-423f-bc88-93711eaec908") : secret "image-registry-tls" not found Oct 30 16:22:42.453508 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.453419 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:22:42.453508 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:42.453442 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs podName:1ab95e94-bdff-4f1f-a0fe-095a2509d6f8 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.453431097 +0000 UTC m=+37.688054725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs") pod "router-default-58c95ddf47-6m2jw" (UID: "1ab95e94-bdff-4f1f-a0fe-095a2509d6f8") : secret "router-metrics-certs-default" not found Oct 30 16:22:42.497196 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:42.497088 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:42.651163 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:42.650524 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rmgmf" event={"ID":"a21d3561-30d9-4a48-816b-5b7be4e5813d","Type":"ContainerStarted","Data":"9ad69069889b3f897c4b7264eec4c937234a4dca555860ae48aa11b96f2a78c5"} Oct 30 16:22:42.694465 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:42.694420 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rmgmf" podStartSLOduration=196.607071121 podStartE2EDuration="3m18.694407163s" totalImagesPullingTime="2.087336042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:42.693736358 +0000 UTC m=+33.928360006" watchObservedRunningTime="2026-10-30 16:22:42.694407163 +0000 UTC m=+33.929030845" Oct 30 16:22:42.849367 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:42.849276 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:43.059757 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:43.059704 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:43.059941 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:43.059841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:43.060532 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:43.060110 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:22:43.060532 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:43.060203 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs podName:cff25808-bc4e-4bd3-b38c-5503a00d45fa nodeName:}" failed. No retries permitted until 2026-10-30 16:23:15.060181428 +0000 UTC m=+66.294805071 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs") pod "network-metrics-daemon-fz7kn" (UID: "cff25808-bc4e-4bd3-b38c-5503a00d45fa") : secret "metrics-daemon-secret" not found Oct 30 16:22:43.073261 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:43.073204 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:43.114107 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:43.114078 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:43.161491 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:43.161453 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmvc\" (UniqueName: \"kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc\") pod \"network-check-target-hptcz\" (UID: \"ddae22b7-ddc1-4766-9af6-3899f1edf48c\") " pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:43.165904 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:43.165874 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jmvc\" (UniqueName: \"kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc\") pod \"network-check-target-hptcz\" (UID: \"ddae22b7-ddc1-4766-9af6-3899f1edf48c\") " pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:43.391956 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:43.391916 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:43.645589 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:43.645503 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pq64k"] Oct 30 16:22:43.656024 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:43.656002 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/0.log" Oct 30 16:22:43.656132 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:43.656043 2574 generic.go:356] "Generic (PLEG): container finished" podID="5a25e3b8-4f22-4fdb-86a9-70d1c594236d" containerID="8eced1c6ad90430be0adeab7541f40a8a945aa81cc403db91425daeea9c0db24" exitCode=255 Oct 30 16:22:43.656871 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:43.656789 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" event={"ID":"5a25e3b8-4f22-4fdb-86a9-70d1c594236d","Type":"ContainerDied","Data":"8eced1c6ad90430be0adeab7541f40a8a945aa81cc403db91425daeea9c0db24"} Oct 30 16:22:43.656871 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:43.656840 2574 scope.go:122] "RemoveContainer" containerID="8eced1c6ad90430be0adeab7541f40a8a945aa81cc403db91425daeea9c0db24" Oct 30 16:22:43.786443 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:43.786400 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hptcz"] Oct 30 16:22:43.795427 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:43.795391 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9bd726_ba59_4c55_838f_e00276ca9bef.slice/crio-ff86a527fde692c3c746dfa3a13f176630535ea74055d35d7336f43a7b5cfeae WatchSource:0}: Error finding container ff86a527fde692c3c746dfa3a13f176630535ea74055d35d7336f43a7b5cfeae: Status 404 returned error can't find the container with id ff86a527fde692c3c746dfa3a13f176630535ea74055d35d7336f43a7b5cfeae Oct 30 16:22:43.837070 ip-10-0-131-179 kubenswrapper[2574]: W1030 16:22:43.837030 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddae22b7_ddc1_4766_9af6_3899f1edf48c.slice/crio-dd4258792ba0ee64ef0efd5c54ba63eb4d9a3086731777341a4373baea17c365 WatchSource:0}: Error finding container dd4258792ba0ee64ef0efd5c54ba63eb4d9a3086731777341a4373baea17c365: Status 404 returned error can't find the container with id dd4258792ba0ee64ef0efd5c54ba63eb4d9a3086731777341a4373baea17c365 Oct 30 16:22:44.663429 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:44.663395 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kx55q" event={"ID":"9c282411-f8cb-4c83-8959-52035df7cd24","Type":"ContainerStarted","Data":"fc994fdddaae903e9e11783366592236bc35dd26c624858f80f5a77cb3594ba6"} Oct 30 16:22:44.665985 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:44.665456 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" event={"ID":"a1e1a5f5-7d62-43a9-9a51-1cb702e80963","Type":"ContainerStarted","Data":"c3fb517d5bc9c0c3d1152a3810d612436d51bbd2dd62ef8de8c9923927db7771"} Oct 30 16:22:44.670340 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:44.670318 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/1.log" Oct 30 16:22:44.670788 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:44.670771 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/0.log" Oct 30 16:22:44.670918 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:44.670803 2574 generic.go:356] "Generic (PLEG): container finished" podID="5a25e3b8-4f22-4fdb-86a9-70d1c594236d" containerID="9327de775d1ac0e11a7137ae4ef9fb291b5a69127a8688d920be8a7a4ccfe5fa" exitCode=255 Oct 30 16:22:44.670918 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:44.670879 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" event={"ID":"5a25e3b8-4f22-4fdb-86a9-70d1c594236d","Type":"ContainerDied","Data":"9327de775d1ac0e11a7137ae4ef9fb291b5a69127a8688d920be8a7a4ccfe5fa"} Oct 30 16:22:44.671141 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:44.670940 2574 scope.go:122] "RemoveContainer" containerID="8eced1c6ad90430be0adeab7541f40a8a945aa81cc403db91425daeea9c0db24" Oct 30 16:22:44.671226 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:44.671205 2574 scope.go:122] "RemoveContainer" containerID="9327de775d1ac0e11a7137ae4ef9fb291b5a69127a8688d920be8a7a4ccfe5fa" Oct 30 16:22:44.671414 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:44.671396 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-szx5s_openshift-console-operator(5a25e3b8-4f22-4fdb-86a9-70d1c594236d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" podUID="5a25e3b8-4f22-4fdb-86a9-70d1c594236d" Oct 30 16:22:44.673477 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:44.673458 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" event={"ID":"a4922dc9-bd15-45a0-a9f6-e61089e6096b","Type":"ContainerStarted","Data":"6c4930a534499963dd70c355a8fb72f0d6cef3a539b72956ce22c8423fc5779c"} Oct 30 16:22:44.675328 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:44.675307 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hptcz" event={"ID":"ddae22b7-ddc1-4766-9af6-3899f1edf48c","Type":"ContainerStarted","Data":"dd4258792ba0ee64ef0efd5c54ba63eb4d9a3086731777341a4373baea17c365"} Oct 30 16:22:44.677196 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:44.677174 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pq64k" event={"ID":"cb9bd726-ba59-4c55-838f-e00276ca9bef","Type":"ContainerStarted","Data":"ff86a527fde692c3c746dfa3a13f176630535ea74055d35d7336f43a7b5cfeae"} Oct 30 16:22:44.709764 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:44.709696 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-kx55q" podStartSLOduration=195.695644376 podStartE2EDuration="3m20.709678698s" totalImagesPullingTime="5.014034322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:44.708574907 +0000 UTC m=+35.943198561" watchObservedRunningTime="2026-10-30 16:22:44.709678698 +0000 UTC m=+35.944302348" Oct 30 16:22:44.851984 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:44.851272 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" podStartSLOduration=195.836830044 podStartE2EDuration="3m20.851251983s" totalImagesPullingTime="5.014421939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:44.850432716 +0000 UTC m=+36.085056365" watchObservedRunningTime="2026-10-30 16:22:44.851251983 +0000 UTC m=+36.085875635" Oct 30 16:22:44.851984 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:44.851381 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" podStartSLOduration=196.413206711 podStartE2EDuration="3m20.85137429s" totalImagesPullingTime="4.438167579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:44.80692513 +0000 UTC m=+36.041548780" watchObservedRunningTime="2026-10-30 16:22:44.85137429 +0000 UTC m=+36.085997941" Oct 30 16:22:44.983667 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:44.983637 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:45.136279 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.136234 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:45.283568 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.283447 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:45.630626 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.630537 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:45.683947 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.683915 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sj6h8" event={"ID":"12874a88-98a6-436e-973d-cb931fc769de","Type":"ContainerStarted","Data":"2a3484ec76c7ebbc9870c35890bed05f60223007624d82dce7370d1d1f88ba54"} Oct 30 16:22:45.686912 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.686700 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/1.log" Oct 30 16:22:45.687068 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.687053 2574 scope.go:122] "RemoveContainer" containerID="9327de775d1ac0e11a7137ae4ef9fb291b5a69127a8688d920be8a7a4ccfe5fa" Oct 30 16:22:45.687230 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:45.687211 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-szx5s_openshift-console-operator(5a25e3b8-4f22-4fdb-86a9-70d1c594236d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" podUID="5a25e3b8-4f22-4fdb-86a9-70d1c594236d" Oct 30 16:22:45.689029 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.689004 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mfwcr" event={"ID":"4dfa9a2f-f22b-4c85-a594-903928b2034a","Type":"ContainerStarted","Data":"4b20af91bf74869711752b5c39667ef7d33bdab5a7e38e878873edbcf940c19c"} Oct 30 16:22:45.782357 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.782301 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sj6h8" podStartSLOduration=193.840289282 podStartE2EDuration="3m19.782281436s" totalImagesPullingTime="5.941992154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:19:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:45.730376074 +0000 UTC m=+36.964999727" watchObservedRunningTime="2026-10-30 16:22:45.782281436 +0000 UTC m=+37.016905087" Oct 30 16:22:45.784285 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.784241 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:45.826235 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.826155 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mfwcr" podStartSLOduration=2.429633027 podStartE2EDuration="7.826140412s" totalImagesPullingTime="5.396507385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:45.825842213 +0000 UTC m=+37.060465866" watchObservedRunningTime="2026-10-30 16:22:45.826140412 +0000 UTC m=+37.060764063" Oct 30 16:22:45.891343 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.891301 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wnmqt"] Oct 30 16:22:45.905865 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.905177 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:45.915947 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.915924 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wnmqt"] Oct 30 16:22:45.921446 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.921425 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 16:22:45.921804 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.921693 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 16:22:45.921921 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.921883 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lpvkt\"" Oct 30 16:22:45.997097 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.997063 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:45.997240 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.997128 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21d827a7-6f18-4be7-86eb-b9141d12a44f-crio-socket\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:45.997240 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.997153 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnbxr\" (UniqueName: \"kubernetes.io/projected/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-api-access-lnbxr\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:45.997240 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.997210 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21d827a7-6f18-4be7-86eb-b9141d12a44f-data-volume\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:45.997454 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:45.997438 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:46.008415 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.008337 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:46.099961 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.098290 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:46.099961 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.098357 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:46.099961 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.098405 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21d827a7-6f18-4be7-86eb-b9141d12a44f-crio-socket\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:46.099961 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.098430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnbxr\" (UniqueName: \"kubernetes.io/projected/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-api-access-lnbxr\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:46.099961 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.098498 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21d827a7-6f18-4be7-86eb-b9141d12a44f-data-volume\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:46.099961 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.098921 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21d827a7-6f18-4be7-86eb-b9141d12a44f-data-volume\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:46.099961 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.099043 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:22:46.099961 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.099109 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls podName:21d827a7-6f18-4be7-86eb-b9141d12a44f nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.599090841 +0000 UTC m=+37.833714469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnmqt" (UID: "21d827a7-6f18-4be7-86eb-b9141d12a44f") : secret "insights-runtime-extractor-tls" not found Oct 30 16:22:46.099961 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.099430 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:22:46.099961 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.099538 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-rbac-proxy-cm podName:21d827a7-6f18-4be7-86eb-b9141d12a44f nodeName:}" failed. No retries permitted until 2026-10-30 16:22:46.599523096 +0000 UTC m=+37.834146739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wnmqt" (UID: "21d827a7-6f18-4be7-86eb-b9141d12a44f") : configmap "kube-rbac-proxy" not found Oct 30 16:22:46.099961 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.099637 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21d827a7-6f18-4be7-86eb-b9141d12a44f-crio-socket\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:46.133554 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.133470 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnbxr\" (UniqueName: \"kubernetes.io/projected/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-api-access-lnbxr\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:46.147864 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.147823 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:46.502282 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.502245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:46.502458 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.502321 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6xx9g\" (UID: \"78ae398a-fdb1-48f8-94fa-51eb582e663a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:46.502458 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.502361 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:46.502458 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.502403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:46.502458 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.502430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:46.502686 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.502466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:46.502686 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.502506 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:46.502686 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.502542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:46.502686 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.502673 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:22:46.502686 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.502686 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8: secret "image-registry-tls" not found Oct 30 16:22:46.502948 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.502766 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls podName:fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:54.50274934 +0000 UTC m=+45.737372983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls") pod "image-registry-5cc8d7fd6b-v2gt8" (UID: "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5") : secret "image-registry-tls" not found Oct 30 16:22:46.503272 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.503110 2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:22:46.503272 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.503123 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:22:46.503272 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.503148 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle podName:1ab95e94-bdff-4f1f-a0fe-095a2509d6f8 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:54.503128965 +0000 UTC m=+45.737752608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle") pod "router-default-58c95ddf47-6m2jw" (UID: "1ab95e94-bdff-4f1f-a0fe-095a2509d6f8") : configmap references non-existent config key: service-ca.crt Oct 30 16:22:46.503272 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.503169 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert podName:78ae398a-fdb1-48f8-94fa-51eb582e663a nodeName:}" failed. No retries permitted until 2026-10-30 16:22:54.503158497 +0000 UTC m=+45.737782126 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6xx9g" (UID: "78ae398a-fdb1-48f8-94fa-51eb582e663a") : secret "networking-console-plugin-cert" not found Oct 30 16:22:46.503272 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.503185 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs podName:1ab95e94-bdff-4f1f-a0fe-095a2509d6f8 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:54.503176933 +0000 UTC m=+45.737800564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs") pod "router-default-58c95ddf47-6m2jw" (UID: "1ab95e94-bdff-4f1f-a0fe-095a2509d6f8") : secret "router-metrics-certs-default" not found Oct 30 16:22:46.503272 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.503168 2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:22:46.503272 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.503212 2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:22:46.503272 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.503218 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls podName:7fff2d07-9f9e-4edb-a32c-ad3a9440fc17 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:54.50320979 +0000 UTC m=+45.737833433 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-btqzf" (UID: "7fff2d07-9f9e-4edb-a32c-ad3a9440fc17") : secret "samples-operator-tls" not found Oct 30 16:22:46.503272 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.503218 2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:46.503272 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.503224 2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dfdc4b7f-lkk6g: secret "image-registry-tls" not found Oct 30 16:22:46.503272 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.503236 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:22:46.503272 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.503253 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls podName:93b8a9b8-43df-4dc0-97da-4bfd11f373cd nodeName:}" failed. No retries permitted until 2026-10-30 16:22:54.503243782 +0000 UTC m=+45.737867423 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gn2hp" (UID: "93b8a9b8-43df-4dc0-97da-4bfd11f373cd") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:46.503272 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.503270 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls podName:7932d7cc-b9fe-423f-bc88-93711eaec908 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:54.50325982 +0000 UTC m=+45.737883460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls") pod "image-registry-5dfdc4b7f-lkk6g" (UID: "7932d7cc-b9fe-423f-bc88-93711eaec908") : secret "image-registry-tls" not found Oct 30 16:22:46.503272 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.503284 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls podName:f79b5c8d-c736-4db0-93b4-b6a40ad58955 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:54.503276783 +0000 UTC m=+45.737900413 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls") pod "dns-default-82w6j" (UID: "f79b5c8d-c736-4db0-93b4-b6a40ad58955") : secret "dns-default-metrics-tls" not found Oct 30 16:22:46.603842 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.603810 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:46.604015 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.603868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:46.604015 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.603970 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:22:46.604015 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.603998 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:22:46.604186 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.604052 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-rbac-proxy-cm podName:21d827a7-6f18-4be7-86eb-b9141d12a44f nodeName:}" failed. No retries permitted until 2026-10-30 16:22:47.604035091 +0000 UTC m=+38.838658732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wnmqt" (UID: "21d827a7-6f18-4be7-86eb-b9141d12a44f") : configmap "kube-rbac-proxy" not found Oct 30 16:22:46.604186 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:46.604071 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls podName:21d827a7-6f18-4be7-86eb-b9141d12a44f nodeName:}" failed. No retries permitted until 2026-10-30 16:22:47.604061557 +0000 UTC m=+38.838685200 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnmqt" (UID: "21d827a7-6f18-4be7-86eb-b9141d12a44f") : secret "insights-runtime-extractor-tls" not found Oct 30 16:22:46.656385 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.656354 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:46.799878 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:46.799788 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:47.390082 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:47.389998 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:47.613157 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:47.613086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:47.613316 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:47.613192 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:47.613316 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:47.613251 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:22:47.613420 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:47.613325 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls podName:21d827a7-6f18-4be7-86eb-b9141d12a44f nodeName:}" failed. No retries permitted until 2026-10-30 16:22:49.613309302 +0000 UTC m=+40.847932954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnmqt" (UID: "21d827a7-6f18-4be7-86eb-b9141d12a44f") : secret "insights-runtime-extractor-tls" not found Oct 30 16:22:47.613420 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:47.613330 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 16:22:47.613420 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:47.613393 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-rbac-proxy-cm podName:21d827a7-6f18-4be7-86eb-b9141d12a44f nodeName:}" failed. No retries permitted until 2026-10-30 16:22:49.613375596 +0000 UTC m=+40.847999226 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wnmqt" (UID: "21d827a7-6f18-4be7-86eb-b9141d12a44f") : configmap "kube-rbac-proxy" not found Oct 30 16:22:47.701974 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:47.701896 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hptcz" event={"ID":"ddae22b7-ddc1-4766-9af6-3899f1edf48c","Type":"ContainerStarted","Data":"b72410f4904fb0e748516b8aa81ea9940b2e02d3f0f173555dd88e036b101281"} Oct 30 16:22:47.702135 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:47.702026 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:47.749655 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:47.749598 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hptcz" podStartSLOduration=35.59999383 podStartE2EDuration="38.749584925s" totalImagesPullingTime="3.149591095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:22:47.749315612 +0000 UTC m=+38.983939262" watchObservedRunningTime="2026-10-30 16:22:47.749584925 +0000 UTC m=+38.984208575" Oct 30 16:22:49.067845 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:49.067817 2574 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:49.068222 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:49.067867 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:49.068222 ip-10-0-131-179 kubenswrapper[2574]: I1030 16:22:49.068160 2574 scope.go:122] "RemoveContainer" containerID="9327de775d1ac0e11a7137ae4ef9fb291b5a69127a8688d920be8a7a4ccfe5fa" Oct 30 16:22:49.068345 ip-10-0-131-179 kubenswrapper[2574]: E1030 16:22:49.068328 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-szx5s_openshift-console-operator(5a25e3b8-4f22-4fdb-86a9-70d1c594236d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" podUID="5a25e3b8-4f22-4fdb-86a9-70d1c594236d" Oct 30 16:22:49.601885 ip-10-0-131-179 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 16:22:49.616136 ip-10-0-131-179 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 16:22:49.616304 ip-10-0-131-179 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 16:22:49.616354 ip-10-0-131-179 systemd[1]: kubelet.service: Consumed 2.981s CPU time. Oct 30 16:22:49.626194 ip-10-0-131-179 systemd[1]: Starting Kubernetes Kubelet... Oct 30 16:22:49.720533 ip-10-0-131-179 kubenswrapper[4861]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:22:49.720533 ip-10-0-131-179 kubenswrapper[4861]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 16:22:49.720533 ip-10-0-131-179 kubenswrapper[4861]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:22:49.720533 ip-10-0-131-179 kubenswrapper[4861]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:22:49.722595 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722518 4861 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:49.722595 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722595 4861 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:49.722595 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722598 4861 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722602 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722605 4861 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722607 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722610 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722613 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722615 4861 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722618 4861 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722620 4861 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722628 4861 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722631 4861 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722634 4861 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722636 4861 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722639 4861 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722641 4861 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722644 4861 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722646 4861 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722649 4861 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722651 4861 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722654 4861 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:49.722692 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722659 4861 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722663 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722666 4861 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722669 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722671 4861 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722674 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722677 4861 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722679 4861 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722682 4861 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722685 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722687 4861 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722690 4861 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722693 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722695 4861 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722698 4861 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722700 4861 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722703 4861 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722706 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722709 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722711 4861 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:49.723209 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722714 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722717 4861 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722736 4861 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722739 4861 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722742 4861 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722745 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722748 4861 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722750 4861 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722752 4861 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722755 4861 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722757 4861 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722760 4861 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722762 4861 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722765 4861 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722770 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722773 4861 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722776 4861 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722778 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722781 4861 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722785 4861 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:49.723689 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722789 4861 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722792 4861 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722796 4861 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722798 4861 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722802 4861 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722807 4861 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722810 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722813 4861 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722816 4861 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722819 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722822 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722825 4861 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722827 4861 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722830 4861 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722833 4861 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722835 4861 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722837 4861 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722840 4861 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722843 4861 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:49.724227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722845 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722848 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722850 4861 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722853 4861 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722856 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722858 4861 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722860 4861 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722865 4861 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722867 4861 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722870 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722872 4861 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722875 4861 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722878 4861 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722880 4861 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722883 4861 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722885 4861 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722888 4861 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722890 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722893 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:49.724688 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722896 4861 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722898 4861 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722900 4861 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722903 4861 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722906 4861 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.722910 4861 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723709 4861 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723716 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723735 4861 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723739 4861 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723741 4861 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723744 4861 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723747 4861 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723749 4861 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723752 4861 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723754 4861 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723758 4861 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723762 4861 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723765 4861 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723768 4861 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:49.725160 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723771 4861 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723774 4861 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723776 4861 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723779 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723781 4861 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723784 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723786 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723789 4861 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723791 4861 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723794 4861 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723796 4861 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723798 4861 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723801 4861 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723804 4861 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723806 4861 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723808 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723811 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723813 4861 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723817 4861 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:49.725673 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723819 4861 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723822 4861 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723824 4861 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723827 4861 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723829 4861 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723831 4861 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723834 4861 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723837 4861 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723840 4861 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723842 4861 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723846 4861 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723849 4861 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723852 4861 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723856 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723858 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723861 4861 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723863 4861 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723866 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723868 4861 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:49.726172 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723871 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723873 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723876 4861 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723878 4861 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723880 4861 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723883 4861 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723885 4861 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723888 4861 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723890 4861 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723893 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723895 4861 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723898 4861 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723901 4861 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723903 4861 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723906 4861 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723910 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723913 4861 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723915 4861 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723917 4861 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723920 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:49.726641 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723923 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723925 4861 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723928 4861 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723930 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723933 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723935 4861 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723938 4861 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723941 4861 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723943 4861 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723946 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723948 4861 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723951 4861 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723954 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723956 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723958 4861 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723961 4861 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723963 4861 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723965 4861 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723968 4861 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:49.727147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723970 4861 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723973 4861 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723975 4861 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723978 4861 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723980 4861 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723982 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723985 4861 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723988 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723991 4861 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723993 4861 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723996 4861 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.723998 4861 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724002 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724004 4861 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724007 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724464 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724469 4861 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724471 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724474 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724477 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:49.727606 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724479 4861 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724482 4861 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724484 4861 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724487 4861 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724489 4861 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724492 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724494 4861 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724497 4861 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724499 4861 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724502 4861 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724504 4861 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724509 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724511 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724514 4861 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724516 4861 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724521 4861 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724525 4861 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724528 4861 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724531 4861 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:49.728101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724534 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724537 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724539 4861 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724542 4861 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724544 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724547 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724549 4861 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724552 4861 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724555 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724557 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724560 4861 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724563 4861 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724565 4861 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724568 4861 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724570 4861 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724573 4861 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724575 4861 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724577 4861 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724580 4861 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724582 4861 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:49.728551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724585 4861 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724587 4861 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724589 4861 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724592 4861 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724596 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724598 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724601 4861 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724604 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724606 4861 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724609 4861 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724611 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724616 4861 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724619 4861 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724627 4861 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724630 4861 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724632 4861 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724635 4861 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724638 4861 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724640 4861 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724642 4861 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:49.729045 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724645 4861 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724648 4861 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724651 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724653 4861 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724655 4861 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724665 4861 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724668 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724670 4861 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724673 4861 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724675 4861 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724678 4861 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724681 4861 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724685 4861 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724688 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724691 4861 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724694 4861 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724698 4861 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724701 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724704 4861 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:49.729572 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724706 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724709 4861 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724711 4861 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724714 4861 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724732 4861 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724735 4861 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724738 4861 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724746 4861 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724749 4861 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724751 4861 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724754 4861 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724756 4861 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724759 4861 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724761 4861 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724764 4861 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724766 4861 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724769 4861 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724771 4861 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724774 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:49.730068 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724776 4861 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724779 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724781 4861 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.724784 4861 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724878 4861 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724886 4861 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724899 4861 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724904 4861 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724908 4861 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724911 4861 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724916 4861 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724920 4861 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724924 4861 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724926 4861 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724930 4861 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724933 4861 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724936 4861 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724940 4861 flags.go:64] FLAG: --cgroup-root="" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724943 4861 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724945 4861 flags.go:64] FLAG: --client-ca-file="" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724948 4861 flags.go:64] FLAG: --cloud-provider="external" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724957 4861 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724963 4861 flags.go:64] FLAG: --cluster-domain="" Oct 30 16:22:49.730575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724966 4861 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724969 4861 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724972 4861 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724976 4861 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724980 4861 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724983 4861 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724986 4861 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724989 4861 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724992 4861 flags.go:64] FLAG: --contention-profiling="false" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724995 4861 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.724998 4861 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725001 4861 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725003 4861 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725008 4861 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725010 4861 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725013 4861 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725016 4861 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725019 4861 flags.go:64] FLAG: --enable-server="true" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725022 4861 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725031 4861 flags.go:64] FLAG: --event-burst="100" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725034 4861 flags.go:64] FLAG: --event-qps="50" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725037 4861 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725040 4861 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725043 4861 flags.go:64] FLAG: --eviction-hard="" Oct 30 16:22:49.731223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725046 4861 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725049 4861 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725053 4861 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725056 4861 flags.go:64] FLAG: --eviction-soft="" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725060 4861 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725062 4861 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725065 4861 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725074 4861 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725077 4861 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725079 4861 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725082 4861 flags.go:64] FLAG: --feature-gates="" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725086 4861 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725089 4861 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725092 4861 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725095 4861 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725098 4861 flags.go:64] FLAG: --healthz-port="10248" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725101 4861 flags.go:64] FLAG: --help="false" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725104 4861 flags.go:64] FLAG: --hostname-override="ip-10-0-131-179.ec2.internal" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725107 4861 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725110 4861 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725112 4861 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725116 4861 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725119 4861 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725122 4861 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 16:22:49.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725125 4861 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725128 4861 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725130 4861 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725135 4861 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725138 4861 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725141 4861 flags.go:64] FLAG: --kube-reserved="" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725144 4861 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725146 4861 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725149 4861 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725152 4861 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725156 4861 flags.go:64] FLAG: --lock-file="" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725158 4861 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725161 4861 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725165 4861 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725170 4861 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725173 4861 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725185 4861 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725188 4861 flags.go:64] FLAG: --logging-format="text" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725191 4861 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725194 4861 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725197 4861 flags.go:64] FLAG: --manifest-url="" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725200 4861 flags.go:64] FLAG: --manifest-url-header="" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725204 4861 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725207 4861 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725211 4861 flags.go:64] FLAG: --max-pods="110" Oct 30 16:22:49.732476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725214 4861 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725217 4861 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725220 4861 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725222 4861 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725226 4861 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725228 4861 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725231 4861 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725238 4861 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725241 4861 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725244 4861 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725248 4861 flags.go:64] FLAG: --pod-cidr="" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725251 4861 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725253 4861 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725256 4861 flags.go:64] FLAG: --pods-per-core="0" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725259 4861 flags.go:64] FLAG: --port="10250" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725262 4861 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725265 4861 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07f6607cfa6e733f5" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725269 4861 flags.go:64] FLAG: --qos-reserved="" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725272 4861 flags.go:64] FLAG: --read-only-port="10255" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725275 4861 flags.go:64] FLAG: --register-node="true" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725277 4861 flags.go:64] FLAG: --register-with-taints="" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725281 4861 flags.go:64] FLAG: --registry-burst="10" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725284 4861 flags.go:64] FLAG: --registry-qps="5" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725287 4861 flags.go:64] FLAG: --reserved-cpus="" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725291 4861 flags.go:64] FLAG: --reserved-memory="" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725294 4861 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 16:22:49.733448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725297 4861 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725300 4861 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725303 4861 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725306 4861 flags.go:64] FLAG: --runonce="false" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725309 4861 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725312 4861 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725314 4861 flags.go:64] FLAG: --seccomp-default="false" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725317 4861 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725320 4861 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725323 4861 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725326 4861 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725329 4861 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725332 4861 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725334 4861 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725337 4861 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725340 4861 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725344 4861 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725347 4861 flags.go:64] FLAG: --system-cgroups="" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725349 4861 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725354 4861 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725357 4861 flags.go:64] FLAG: --tls-cert-file="" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725360 4861 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725368 4861 flags.go:64] FLAG: --tls-min-version="" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725373 4861 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725376 4861 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 16:22:49.734519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725378 4861 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725381 4861 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725384 4861 flags.go:64] FLAG: --v="2" Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725388 4861 flags.go:64] FLAG: --version="false" Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725392 4861 flags.go:64] FLAG: --vmodule="" Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725396 4861 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725400 4861 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725542 4861 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725546 4861 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725549 4861 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725551 4861 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725554 4861 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725556 4861 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725559 4861 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725561 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725564 4861 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725567 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725569 4861 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725572 4861 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725574 4861 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725577 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:49.735605 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725579 4861 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725582 4861 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725585 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725588 4861 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725591 4861 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725593 4861 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725595 4861 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725598 4861 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725600 4861 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725604 4861 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725606 4861 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725609 4861 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725612 4861 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725614 4861 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725617 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725619 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725622 4861 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725624 4861 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725627 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725629 4861 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:49.736386 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725632 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725634 4861 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725637 4861 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725639 4861 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725642 4861 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725644 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725647 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725649 4861 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725652 4861 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725655 4861 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725657 4861 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725660 4861 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725662 4861 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725664 4861 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725668 4861 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725670 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725673 4861 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725676 4861 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725678 4861 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725681 4861 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:49.737121 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725684 4861 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725687 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725690 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725693 4861 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725695 4861 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725698 4861 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725700 4861 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725702 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725705 4861 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725708 4861 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725710 4861 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725713 4861 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725715 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725731 4861 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725735 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725737 4861 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725739 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725743 4861 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725747 4861 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:49.737742 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725750 4861 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725752 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725755 4861 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725757 4861 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725760 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725762 4861 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725765 4861 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725768 4861 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725771 4861 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725774 4861 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725776 4861 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725779 4861 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725781 4861 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725784 4861 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725788 4861 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725790 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725793 4861 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725796 4861 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725798 4861 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:49.738208 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725800 4861 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:49.738950 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725805 4861 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:49.738950 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725809 4861 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:49.738950 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725812 4861 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:49.738950 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725815 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:49.738950 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725818 4861 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:49.738950 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725821 4861 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:49.738950 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725823 4861 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:49.738950 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725826 4861 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:49.738950 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725829 4861 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:49.738950 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725831 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:49.738950 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725833 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:49.738950 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725836 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:49.738950 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725838 4861 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:49.738950 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.725863 4861 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:22:49.738950 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725897 4861 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:49.738950 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725901 4861 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725904 4861 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725907 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725910 4861 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725913 4861 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725919 4861 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725921 4861 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725924 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725927 4861 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725929 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725932 4861 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725936 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725939 4861 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725943 4861 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725946 4861 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725949 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725951 4861 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725954 4861 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725956 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:49.739472 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725959 4861 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725962 4861 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725964 4861 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725967 4861 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725970 4861 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725972 4861 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725975 4861 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725978 4861 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725980 4861 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725982 4861 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725985 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725987 4861 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725990 4861 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725992 4861 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.725995 4861 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726000 4861 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726003 4861 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726007 4861 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726011 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:49.740123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726013 4861 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726015 4861 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726018 4861 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726021 4861 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726023 4861 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726027 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726029 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726032 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726034 4861 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726037 4861 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726039 4861 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726041 4861 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726044 4861 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726046 4861 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726049 4861 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726051 4861 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726054 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726056 4861 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726059 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726061 4861 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:49.740586 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726064 4861 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726066 4861 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726068 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726071 4861 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726073 4861 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726076 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726078 4861 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726080 4861 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726083 4861 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726085 4861 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726087 4861 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726091 4861 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726094 4861 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726096 4861 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726099 4861 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726101 4861 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726104 4861 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726108 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726110 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726113 4861 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:49.741102 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726115 4861 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726117 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726120 4861 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726123 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726125 4861 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726127 4861 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726130 4861 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726132 4861 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726135 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726137 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726140 4861 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726143 4861 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726146 4861 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726148 4861 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726152 4861 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726154 4861 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726156 4861 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726159 4861 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726161 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:49.741597 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726163 4861 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726166 4861 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726168 4861 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726170 4861 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726174 4861 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726176 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726179 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.726182 4861 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.729913 4861 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.729926 4861 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.729943 4861 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.729947 4861 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730004 4861 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730008 4861 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730012 4861 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730015 4861 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730018 4861 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730020 4861 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730023 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730026 4861 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:49.742106 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730029 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730031 4861 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730034 4861 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730036 4861 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730040 4861 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730043 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730046 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730048 4861 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730051 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730053 4861 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730056 4861 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730059 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730062 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730065 4861 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730067 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730070 4861 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730073 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730075 4861 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730078 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:49.742604 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730080 4861 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730084 4861 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730087 4861 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730089 4861 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730092 4861 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730094 4861 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730097 4861 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730099 4861 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730102 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730105 4861 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730107 4861 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730110 4861 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730112 4861 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730115 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730117 4861 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730119 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730122 4861 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730125 4861 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730127 4861 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:49.743127 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730130 4861 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730133 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730135 4861 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730138 4861 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730140 4861 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730143 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730146 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730148 4861 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730151 4861 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730154 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730156 4861 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730159 4861 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730161 4861 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730164 4861 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730166 4861 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730169 4861 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730171 4861 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730174 4861 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730176 4861 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730179 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:49.743579 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730181 4861 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730183 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730186 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730189 4861 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730191 4861 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730193 4861 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730195 4861 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730198 4861 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730200 4861 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730203 4861 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730205 4861 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730208 4861 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730211 4861 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730213 4861 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730216 4861 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730218 4861 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730221 4861 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730225 4861 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730229 4861 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730233 4861 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:49.744073 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730235 4861 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730238 4861 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730240 4861 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730244 4861 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730248 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730251 4861 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730255 4861 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730257 4861 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730260 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730263 4861 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730266 4861 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730269 4861 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730271 4861 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730274 4861 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730276 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730279 4861 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730281 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730284 4861 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730287 4861 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:49.744573 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730289 4861 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:49.745193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.730315 4861 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:22:49.745193 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730446 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:49.745193 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730450 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:49.745193 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730453 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:49.745193 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730456 4861 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:49.745193 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730461 4861 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:49.745193 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730464 4861 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:49.745193 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730467 4861 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:49.745193 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730470 4861 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:49.745193 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730473 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:49.745193 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730475 4861 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:49.745193 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730478 4861 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:49.745193 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730480 4861 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:49.745193 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730483 4861 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:49.745193 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730485 4861 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:49.745193 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730488 4861 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730490 4861 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730493 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730495 4861 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730497 4861 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730500 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730502 4861 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730505 4861 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730507 4861 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730510 4861 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730512 4861 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730515 4861 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730517 4861 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730519 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730522 4861 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730524 4861 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730528 4861 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730531 4861 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730534 4861 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730537 4861 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:49.745706 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730540 4861 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730543 4861 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730546 4861 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730549 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730551 4861 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730553 4861 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730556 4861 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730558 4861 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730561 4861 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730563 4861 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730565 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730568 4861 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730570 4861 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730573 4861 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730575 4861 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730578 4861 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730581 4861 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730583 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730586 4861 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730589 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:49.746344 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730591 4861 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730594 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730597 4861 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730599 4861 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730602 4861 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730604 4861 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730606 4861 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730609 4861 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730611 4861 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730613 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730615 4861 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730618 4861 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730621 4861 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730623 4861 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730625 4861 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730628 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730631 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730634 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730636 4861 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:49.746974 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730638 4861 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730641 4861 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730643 4861 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730645 4861 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730648 4861 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730651 4861 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730653 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730655 4861 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730658 4861 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730660 4861 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730663 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730665 4861 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730668 4861 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730670 4861 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730672 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730675 4861 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730677 4861 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730679 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730682 4861 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730684 4861 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:49.747583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730686 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:49.748245 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730689 4861 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:49.748245 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730691 4861 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:49.748245 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730695 4861 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:49.748245 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730698 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:49.748245 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730702 4861 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:49.748245 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730704 4861 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:49.748245 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730707 4861 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:49.748245 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730709 4861 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:49.748245 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730711 4861 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:49.748245 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730714 4861 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:49.748245 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730717 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:49.748245 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730736 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:49.748245 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.730766 4861 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:22:49.748245 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730800 4861 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:22:49.748245 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730804 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730807 4861 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730810 4861 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730812 4861 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730814 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730817 4861 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730819 4861 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730822 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730824 4861 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730826 4861 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730829 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730831 4861 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730834 4861 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730837 4861 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730841 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730844 4861 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730847 4861 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730849 4861 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730851 4861 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:22:49.748634 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730854 4861 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730856 4861 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730859 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730862 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730881 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730884 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730887 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730890 4861 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730893 4861 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730896 4861 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730898 4861 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730901 4861 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730903 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730906 4861 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730908 4861 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730911 4861 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730913 4861 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730916 4861 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730918 4861 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730921 4861 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:22:49.749113 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730923 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730934 4861 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730937 4861 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730939 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730942 4861 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730944 4861 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730947 4861 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730949 4861 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730952 4861 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730954 4861 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730957 4861 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730959 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730961 4861 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730964 4861 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730968 4861 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730971 4861 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730975 4861 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730978 4861 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730981 4861 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:22:49.749610 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730983 4861 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730985 4861 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730989 4861 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730992 4861 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730994 4861 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730997 4861 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.730999 4861 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731002 4861 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731004 4861 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731007 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731009 4861 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731012 4861 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731014 4861 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731017 4861 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731019 4861 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731024 4861 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731026 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731029 4861 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731032 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731034 4861 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:22:49.750301 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731038 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731040 4861 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731042 4861 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731045 4861 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731048 4861 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731050 4861 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731053 4861 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731055 4861 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731058 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731060 4861 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731063 4861 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731065 4861 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731068 4861 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731070 4861 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731073 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731075 4861 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731078 4861 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731081 4861 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731084 4861 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:22:49.751147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731087 4861 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:22:49.751831 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731090 4861 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:22:49.751831 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731092 4861 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:22:49.751831 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731094 4861 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:22:49.751831 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731097 4861 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:22:49.751831 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731099 4861 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:22:49.751831 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731102 4861 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:22:49.751831 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:49.731104 4861 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:22:49.751831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.731155 4861 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 16:22:49.751831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.732094 4861 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 16:22:49.751831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.732149 4861 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 16:22:49.751831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.732321 4861 server.go:1014] "Starting client certificate rotation" Oct 30 16:22:49.751831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.732495 4861 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:22:49.751831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.732602 4861 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:17:09 +0000 UTC" deadline="2028-03-30 09:08:53.520817449 +0000 UTC" Oct 30 16:22:49.751831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.732651 4861 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12400h46m3.788170858s" Oct 30 16:22:49.751831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.733603 4861 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:22:49.751831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.735468 4861 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:22:49.752262 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.736696 4861 log.go:25] "Validated CRI v1 runtime API" Oct 30 16:22:49.752262 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.738550 4861 log.go:25] "Validated CRI v1 image API" Oct 30 16:22:49.752262 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.739365 4861 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 16:22:49.752262 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.741261 4861 fs.go:135] Filesystem UUIDs: map[15dd6077-0f08-4130-8124-b9158be1e947:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 acdcf5f9-8141-4a06-8e30-088962adbb5c:/dev/nvme0n1p3] Oct 30 16:22:49.752400 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.741282 4861 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/065bc8e2e2cc3204f157f053f695f5c1b35a88bd84faa84ae93555c2a70cbf9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/065bc8e2e2cc3204f157f053f695f5c1b35a88bd84faa84ae93555c2a70cbf9a/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/303048ebdd7310eda3df6c6f626aa2573535edeb330043ab110100efe677f7cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/303048ebdd7310eda3df6c6f626aa2573535edeb330043ab110100efe677f7cd/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36e6fc788aeb4842a93d5bf9647b4568b11f62808fd1e38e44f7afac45ef82f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36e6fc788aeb4842a93d5bf9647b4568b11f62808fd1e38e44f7afac45ef82f8/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e4c9fa87eb934f94cf97a58acb7e5b132e16fbf5bb86e6cae7cd11327af46f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e4c9fa87eb934f94cf97a58acb7e5b132e16fbf5bb86e6cae7cd11327af46f7/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/63258f8280423ecaf9e3d9bb769a70c1790759e889c97450552d9dcebf05b231/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/63258f8280423ecaf9e3d9bb769a70c1790759e889c97450552d9dcebf05b231/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ceea6bbf77a56ab19647fffef40a9c6dfec70a770767769ce9b22bf5d03d0a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ceea6bbf77a56ab19647fffef40a9c6dfec70a770767769ce9b22bf5d03d0a2/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b233a8a715e2f102d61d0b92dedf99637994546b6ecf815e5702e562c961ce6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b233a8a715e2f102d61d0b92dedf99637994546b6ecf815e5702e562c961ce6/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7cb6b0bd449601611dce1c3efc81c137adc82a59ad929bef2538781be2268c14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7cb6b0bd449601611dce1c3efc81c137adc82a59ad929bef2538781be2268c14/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8538278266c801cf13a88f5be3d63f71f57679c2ccdf4d91f7f9eaa542bdcf8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8538278266c801cf13a88f5be3d63f71f57679c2ccdf4d91f7f9eaa542bdcf8b/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e259c703a3d326d9f791f89f8113b7d78f535765ec289b9b8484e171e9b7198/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e259c703a3d326d9f791f89f8113b7d78f535765ec289b9b8484e171e9b7198/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94e9f3c857ebe0d4f5f59a923374a2f4f52bab39e81de3fa693780477ec79840/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94e9f3c857ebe0d4f5f59a923374a2f4f52bab39e81de3fa693780477ec79840/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1174aee1f9265a7f668bc519d01c5a5f4cfe4965464c6b18b06f69a16219da3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1174aee1f9265a7f668bc519d01c5a5f4cfe4965464c6b18b06f69a16219da3/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aafa2232431d7aa08a6ec916db80c0e6faef13bb2d75d93016de45d34f48c0db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aafa2232431d7aa08a6ec916db80c0e6faef13bb2d75d93016de45d34f48c0db/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8df4a6db68add4fac3ed4114f9358268d5b346bf3b985632487b098d33f2034/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8df4a6db68add4fac3ed4114f9358268d5b346bf3b985632487b098d33f2034/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6646d0f03b00475dcec78da3fdc2ce91bd173d2146c8e0553cfbfd8922b105f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6646d0f03b00475dcec78da3fdc2ce91bd173d2146c8e0553cfbfd8922b105f/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cbfd4442672bbde81d226c4cfbbbefbfad195fe1558d970bf1ae6d5f772c4cbc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cbfd4442672bbde81d226c4cfbbbefbfad195fe1558d970bf1ae6d5f772c4cbc/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9fd7b99fb365a96dc6069ed563255afadfd76487c90447756c14152518ff8bb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9fd7b99fb365a96dc6069ed563255afadfd76487c90447756c14152518ff8bb/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd4258792ba0ee64ef0efd5c54ba63eb4d9a3086731777341a4373baea17c365/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd4258792ba0ee64ef0efd5c54ba63eb4d9a3086731777341a4373baea17c365/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de4bce0ebcb92f776a9467c258d4736af251ff8a1de64fb34ae08714409dd17f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de4bce0ebcb92f776a9467c258d4736af251ff8a1de64fb34ae08714409dd17f/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef582cf82ab8be0925f7cfafdf970d224a83c43f40197f79259d1143549e8226/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef582cf82ab8be0925f7cfafdf970d224a83c43f40197f79259d1143549e8226/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5967c9990da73ecc128bfd5f29740b925371d8923b2b5015f791f1d88ef1905/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5967c9990da73ecc128bfd5f29740b925371d8923b2b5015f791f1d88ef1905/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff86a527fde692c3c746dfa3a13f176630535ea74055d35d7336f43a7b5cfeae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff86a527fde692c3c746dfa3a13f176630535ea74055d35d7336f43a7b5cfeae/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1026393f-ba8e-408a-b7df-a1d03e047b1f/volumes/kubernetes.io~projected/kube-api-access-nwpb8:{mountpoint:/var/lib/kubelet/pods/1026393f-ba8e-408a-b7df-a1d03e047b1f/volumes/kubernetes.io~projected/kube-api-access-nwpb8 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12874a88-98a6-436e-973d-cb931fc769de/volumes/kubernetes.io~projected/kube-api-access-8q6vt:{mountpoint:/var/lib/kubelet/pods/12874a88-98a6-436e-973d-cb931fc769de/volumes/kubernetes.io~projected/kube-api-access-8q6vt major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8/volumes/kubernetes.io~projected/kube-api-access-4qglw:{mountpoint:/var/lib/kubelet/pods/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8/volumes/kubernetes.io~projected/kube-api-access-4qglw major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8/volumes/kubernetes.io~secret/default-certificate major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8/volumes/kubernetes.io~secret/stats-auth major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d0df1a2-2140-46d5-8662-1148296fb5a8/volumes/kubernetes.io~projected/kube-api-access-gzxpj:{mountpoint:/var/lib/kubelet/pods/1d0df1a2-2140-46d5-8662-1148296fb5a8/volumes/kubernetes.io~projected/kube-api-access-gzxpj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21d827a7-6f18-4be7-86eb-b9141d12a44f/volumes/kubernetes.io~projected/kube-api-access-lnbxr:{mountpoint:/var/lib/kubelet/pods/21d827a7-6f18-4be7-86eb-b9141d12a44f/volumes/kubernetes.io~projected/kube-api-access-lnbxr major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0/volumes/kubernetes.io~projected/kube-api-access-lsjwr:{mountpoint:/var/lib/kubelet/pods/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0/volumes/kubernetes.io~projected/kube-api-access-lsjwr major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49aa267f-cf95-4d9b-98c4-df106fcd9620/volumes/kubernetes.io~projected/kube-api-access-nf7x5:{mountpoint:/var/lib/kubelet/pods/49aa267f-cf95-4d9b-98c4-df106fcd9620/volumes/kubernetes.io~projected/kube-api-access-nf7x5 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dfa9a2f-f22b-4c85-a594-903928b2034a/volumes/kubernetes.io~projected/kube-api-access-dcgvd:{mountpoint:/var/lib/kubelet/pods/4dfa9a2f-f22b-4c85-a594-903928b2034a/volumes/kubernetes.io~projected/kube-api-access-dcgvd major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4dfa9a2f-f22b-4c85-a594-903928b2034a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/4dfa9a2f-f22b-4c85-a594-903928b2034a/volumes/kubernetes.io~secret/cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58c9bbd6-243d-4769-afc9-19e5e4097ef9/volumes/kubernetes.io~projected/kube-api-access-tttc4:{mountpoint:/var/lib/kubelet/pods/58c9bbd6-243d-4769-afc9-19e5e4097ef9/volumes/kubernetes.io~projected/kube-api-access-tttc4 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a25e3b8-4f22-4fdb-86a9-70d1c594236d/volumes/kubernetes.io~projected/kube-api-access-fc6nb:{mountpoint:/var/lib/kubelet/pods/5a25e3b8-4f22-4fdb-86a9-70d1c594236d/volumes/kubernetes.io~projected/kube-api-access-fc6nb major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a25e3b8-4f22-4fdb-86a9-70d1c594236d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5a25e3b8-4f22-4fdb-86a9-70d1c594236d/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6089204f-1338-4875-b311-2fa1b27a03d5/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/6089204f-1338-4875-b311-2fa1b27a03d5/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f9aa963-3cff-41d4-925c-79ba5db70d57/volumes/kubernetes.io~projected/kube-api-access-9xlkr:{mountpoint:/var/lib/kubelet/pods/6f9aa963-3cff-41d4-925c-79ba5db70d57/volumes/kubernetes.io~projected/kube-api-access-9xlkr major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7932d7cc-b9fe-423f-bc88-93711eaec908/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/7932d7cc-b9fe-423f-bc88-93711eaec908/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7932d7cc-b9fe-423f-bc88-93711eaec908/volumes/kubernetes.io~projected/kube-api-access-kghb4:{mountpoint:/var/lib/kubelet/pods/7932d7cc-b9fe-423f-bc88-93711eaec908/volumes/kubernetes.io~projected/kube-api-access-kghb4 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7932d7cc-b9fe-423f-bc88-93711eaec908/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/7932d7cc-b9fe-423f-bc88-93711eaec908/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7932d7cc-b9fe-423f-bc88-93711eaec908/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/7932d7cc-b9fe-423f-bc88-93711eaec908/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17/volumes/kubernetes.io~projected/kube-api-access-t929j:{mountpoint:/var/lib/kubelet/pods/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17/volumes/kubernetes.io~projected/kube-api-access-t929j major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93b8a9b8-43df-4dc0-97da-4bfd11f373cd/volumes/kubernetes.io~projected/kube-api-access-66fd7:{mountpoint:/var/lib/kubelet/pods/93b8a9b8-43df-4dc0-97da-4bfd11f373cd/volumes/kubernetes.io~projected/kube-api-access-66fd7 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be/volumes/kubernetes.io~projected/kube-api-access-f9pp4:{mountpoint:/var/lib/kubelet/pods/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be/volumes/kubernetes.io~projected/kube-api-access-f9pp4 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c282411-f8cb-4c83-8959-52035df7cd24/volumes/kubernetes.io~projected/kube-api-access-cgr65:{mountpoint:/var/lib/kubelet/pods/9c282411-f8cb-4c83-8959-52035df7cd24/volumes/kubernetes.io~projected/kube-api-access-cgr65 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c282411-f8cb-4c83-8959-52035df7cd24/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9c282411-f8cb-4c83-8959-52035df7cd24/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1e1a5f5-7d62-43a9-9a51-1cb702e80963/volumes/kubernetes.io~projected/kube-api-access-n2zxc:{mountpoint:/var/lib/kubelet/pods/a1e1a5f5-7d62-43a9-9a51-1cb702e80963/volumes/kubernetes.io~projected/kube-api-access-n2zxc major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1e1a5f5-7d62-43a9-9a51-1cb702e80963/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a1e1a5f5-7d62-43a9-9a51-1cb702e80963/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a21d3561-30d9-4a48-816b-5b7be4e5813d/volumes/kubernetes.io~projected/kube-api-access-lqhlw:{mountpoint:/var/lib/kubelet/pods/a21d3561-30d9-4a48-816b-5b7be4e5813d/volumes/kubernetes.io~projected/kube-api-access-lqhlw major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4922dc9-bd15-45a0-a9f6-e61089e6096b/volumes/kubernetes.io~projected/kube-api-access-ldgcm:{mountpoint:/var/lib/kubelet/pods/a4922dc9-bd15-45a0-a9f6-e61089e6096b/volumes/kubernetes.io~projected/kube-api-access-ldgcm major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4922dc9-bd15-45a0-a9f6-e61089e6096b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a4922dc9-bd15-45a0-a9f6-e61089e6096b/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7e4f934-2451-4133-aa1a-ee1145ac2c41/volumes/kubernetes.io~projected/kube-api-access-64mgb:{mountpoint:/var/lib/kubelet/pods/a7e4f934-2451-4133-aa1a-ee1145ac2c41/volumes/kubernetes.io~projected/kube-api-access-64mgb major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f/volumes/kubernetes.io~projected/kube-api-access-vtkhz:{mountpoint:/var/lib/kubelet/pods/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f/volumes/kubernetes.io~projected/kube-api-access-vtkhz major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8ddcddb-5053-4950-bc4e-5d8f83ec8f08/volumes/kubernetes.io~projected/kube-api-access-qh7x7:{mountpoint:/var/lib/kubelet/pods/c8ddcddb-5053-4950-bc4e-5d8f83ec8f08/volumes/kubernetes.io~projected/kube-api-access-qh7x7 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb9bd726-ba59-4c55-838f-e00276ca9bef/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/cb9bd726-ba59-4c55-838f-e00276ca9bef/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cff25808-bc4e-4bd3-b38c-5503a00d45fa/volumes/kubernetes.io~projected/kube-api-access-rt58k:{mountpoint:/var/lib/kubelet/pods/cff25808-bc4e-4bd3-b38c-5503a00d45fa/volumes/kubernetes.io~projected/kube-api-access-rt58k major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddae22b7-ddc1-4766-9af6-3899f1edf48c/volumes/kubernetes.io~projected/kube-api-access-2jmvc:{mountpoint:/var/lib/kubelet/pods/ddae22b7-ddc1-4766-9af6-3899f1edf48c/volumes/kubernetes.io~projected/kube-api-access-2jmvc major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed9c43e8-604b-44f5-a20c-a3af86848c2c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/ed9c43e8-604b-44f5-a20c-a3af86848c2c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed9c43e8-604b-44f5-a20c-a3af86848c2c/volumes/kubernetes.io~projected/kube-api-access-ngkcw:{mountpoint:/var/lib/kubelet/pods/ed9c43e8-604b-44f5-a20c-a3af86848c2c/volumes/kubernetes.io~projected/kube-api-access-ngkcw major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed9c43e8-604b-44f5-a20c-a3af86848c2c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/ed9c43e8-604b-44f5-a20c-a3af86848c2c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f79b5c8d-c736-4db0-93b4-b6a40ad58955/volumes/kubernetes.io~projected/kube-api-access-gxmwp:{mountpoint:/var/lib/kubelet/pods/f79b5c8d-c736-4db0-93b4-b6a40ad58955/volumes/kubernetes.io~projected/kube-api-access-gxmwp major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5/volumes/kubernetes.io~projected/kube-api-access-zt2sw:{mountpoint:/var/lib/kubelet/pods/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5/volumes/kubernetes.io~projected/kube-api-access-zt2sw major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a8a1f389d736c963380c3013bdd95b7689b92a18ab5bbaa5b40bb3511369ed2a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2417e10214b89ca4bb47d0711ef164aed41b6997c86a4f2778bc0fd18b8a54f3/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/02d34d3747e3122d4ccb486313f8f5c1ffda4d0f2414ab3a40bb52038959bfa0/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/c03a6b5b2b626f3a7b54c57b6ea2d29b93ae09e46adfea62074e5e114986f38e/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/39ba1de5d16704e7bc7bce39df97c34863daf67a9ce409176a74e2b0990c2966/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/abfa4fe185294f38bb153d241e4f200b4634f947b0ec55a14a00a892c80a4916/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5a7c67bfe91cf3efb00ae16f6411df97d5504d37b8d36a2ffb535cc9bc42590a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/95fef74702854b704fdf3eac6bd4fb5a89d61a88d41ea132db2b5220c1490063/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/1726bea704d59e75ac4e4ee490c6d23287b6751cb1a38fe85ed12fd408e4965f/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/7834a2c8f300ac2da2adf3b4e6826433ed538d819e6969cdf1510a10b9d88b78/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/e36c033d0194f97af8f0699f2b72b611821c0ab75da592d87acdfac53093603c/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/9c2136669cca09ff4a23ae9361317acb9165788ab2b672468db738772c63e575/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e374f8bb86191d4a3c6cbebbdc04e796a80859676cad5a93e8d7e237e5b62a2a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/25d1c6f1ecfdae04b6238f2130bfe30b616d4db93e7a5d6adf530a8f2eda4f50/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f66fccdada8b848712cf1eaae81fbd3e1c9473cc16a6055d663e655b14b9d6b2/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/708d917a54e2550e446b988d61d3ac17d5f2245c7ec4485809a2d2bf3971e44c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/e71b23b357ba9ca3f2690668b08f5bf843ee67df292bc1d315ecc88bad84a65b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0e3d8614f7504f29ac5dccedb1d02af3f90f003da9715ec521144a058c4b907a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/6efa8f3d60e713dcefa4a2d577cb315cf044933841927e4e3e29ef5104703281/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f2abd968b350211125730c7948b28102fd7b15b613504d27ed5e2e30b7903eb6/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/523f01eeac8ea7712c653d10e9b2976e403b693856c49b70c3c6e930f08af14c/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/34689028f12cd91e983e7cc5e39078b6ef57d34e6974d511228696359c3eaf89/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/21cc45d30da995f40abb69c8ada03b34fa6ded56ac738901052016e1b5cf537c/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/8dc092162770cad3b1c7affd5187eaeedfeb6bb7efbb69a63a75b447aca54112/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/3711b566041171bf7c1ba95fd4250002355b707f4133612afa3e8451160e5540/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/07b338c769fcaf7301cbd5aff777ff5dc4f4a56c121739ed2c4010bd93b70af1/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/08e170a6853fe44eaf61a90d6168b85142818e7ad041dca984ef39fa512de5fc/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/f377400079799b79fb208cb91be892c113a638796ad4a880b75642eff45f4ec6/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/18dbb0add959e8b1d6e54802ae2d9b85af47646deae802a65244e50f3ecd9352/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/922ee30fe13d8b2a0e0ebc5e14c8c60ea9ade9b374e1b60103f14f7f2a096f1a/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/6d35ca894f9968d5426fa8c0871501a5a52cdf1d49568fc8aefc1371f4f6a392/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/402e76be16cc2d6ee2d366151dc5340789bb1c869e7229ea64dea55aa361e7e9/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/dd392b7c76f34f65357d26f4944cbfacabed242975349fa7a26a92940fbdf58e/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/cfaa5874bf5e583090f0c7d1f11616d2b92c16eb0c981ca9c9f7936cda9bb339/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/3135008708ea9a85496b108a9d57a6eecb459cabf262b2ec24052aeeb96625d3/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/125bddb53498d6e9f692c143568cc509b30641a3d6c2e2c06b9cc87a7caf637c/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/13203d1ae1177d6f52addf6604f660f226a4037cd800e9f8c26ebbe16a3bbf96/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/86916e1287a05528a97008bca5a541fbe4b83e0420ee8f98d83eb878cd79a996/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/e79378b3b82a412db5752225c50d3d2b866b73aab6e2419bbefddbaec8a068cd/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/4f7dcc7a7a838cdb088fe0254f7c670b28ededbd0c71a68b3133ba24d1bf5185/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/d5d2adbbe19a36ea77b1b6df64052ed04d86f6879221448ed511e73effa34a81/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/197301f71f1e46c620b1e5b687937b50f550d38d5b8719143e08b57a0e9e6146/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/48cc2738551c91487cfab03310a6b8b5731d92240c9713ceeea43bad83336524/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/81234d826b515608f66aa856687b70349a35ad0af7003021ae0bb7293afa7fbf/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/d1b83776e0379ab3b71ca08e1d696a5e589473b38223d813638af85905be9e3e/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/956d9b6cb7f2bbf701b862c29439e52166e18a875c35c7f40392d66f51dcebcb/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/a3e2da8acbfdc29aaee14cd226a7fdbb79ec44d8e718f4dfc5f68051f08f63bd/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/a06250d2d7ab9e063445bdf05d93aae0e891f10c08e17aaa1249818633ce52f9/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/a081c61aec442f19d4fc51fe4459f4dd3e1bd77bb363962cde1adf974a2fa403/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/36ebebb7861dc22c4cc9bb8c1f918f7b741eeab469fa53b782338be36195848f/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/90a2c47d6c692244c380b8d78d9f6e2814b4c3e97ee78d5f1f95b37aff4e9362/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/40b7dad8b3441297577141b37206dd330043dc3d3df706f54301b378179f5b79/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/592fa5106d7cc91c88ef6b5d0e5ed6bb8d49b7cb40af85353e0d38c8e502e7bd/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/55b34c5c47822070fb548d47390c6ed9102a506dd75ac8354b9d1b934e540557/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.748541 4861 manager.go:217] Machine: {Timestamp:2026-10-30 16:22:49.748087243 +0000 UTC m=+0.090761272 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102407 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2579b3aac5eceb265019f941b958b6 SystemUUID:ec2579b3-aac5-eceb-2650-19f941b958b6 BootID:f54f2b89-7f37-40e5-83ed-eec2fe0963d7 Filesystems:[{Device:/var/lib/kubelet/pods/1d0df1a2-2140-46d5-8662-1148296fb5a8/volumes/kubernetes.io~projected/kube-api-access-gzxpj DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed9c43e8-604b-44f5-a20c-a3af86848c2c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a4922dc9-bd15-45a0-a9f6-e61089e6096b/volumes/kubernetes.io~projected/kube-api-access-ldgcm DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cb9bd726-ba59-4c55-838f-e00276ca9bef/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff86a527fde692c3c746dfa3a13f176630535ea74055d35d7336f43a7b5cfeae/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7932d7cc-b9fe-423f-bc88-93711eaec908/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5a25e3b8-4f22-4fdb-86a9-70d1c594236d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/12874a88-98a6-436e-973d-cb931fc769de/volumes/kubernetes.io~projected/kube-api-access-8q6vt DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7cb6b0bd449601611dce1c3efc81c137adc82a59ad929bef2538781be2268c14/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b233a8a715e2f102d61d0b92dedf99637994546b6ecf815e5702e562c961ce6/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cbfd4442672bbde81d226c4cfbbbefbfad195fe1558d970bf1ae6d5f772c4cbc/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a4922dc9-bd15-45a0-a9f6-e61089e6096b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8/volumes/kubernetes.io~projected/kube-api-access-4qglw DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/63258f8280423ecaf9e3d9bb769a70c1790759e889c97450552d9dcebf05b231/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7932d7cc-b9fe-423f-bc88-93711eaec908/volumes/kubernetes.io~projected/kube-api-access-kghb4 DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ddae22b7-ddc1-4766-9af6-3899f1edf48c/volumes/kubernetes.io~projected/kube-api-access-2jmvc DeviceMajor:0 DeviceMinor:293 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0/volumes/kubernetes.io~projected/kube-api-access-lsjwr DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de4bce0ebcb92f776a9467c258d4736af251ff8a1de64fb34ae08714409dd17f/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5a25e3b8-4f22-4fdb-86a9-70d1c594236d/volumes/kubernetes.io~projected/kube-api-access-fc6nb DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1e1a5f5-7d62-43a9-9a51-1cb702e80963/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a21d3561-30d9-4a48-816b-5b7be4e5813d/volumes/kubernetes.io~projected/kube-api-access-lqhlw DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8df4a6db68add4fac3ed4114f9358268d5b346bf3b985632487b098d33f2034/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ceea6bbf77a56ab19647fffef40a9c6dfec70a770767769ce9b22bf5d03d0a2/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6089204f-1338-4875-b311-2fa1b27a03d5/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7932d7cc-b9fe-423f-bc88-93711eaec908/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cff25808-bc4e-4bd3-b38c-5503a00d45fa/volumes/kubernetes.io~projected/kube-api-access-rt58k DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/58c9bbd6-243d-4769-afc9-19e5e4097ef9/volumes/kubernetes.io~projected/kube-api-access-tttc4 DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4dfa9a2f-f22b-4c85-a594-903928b2034a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1174aee1f9265a7f668bc519d01c5a5f4cfe4965464c6b18b06f69a16219da3/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f/volumes/kubernetes.io~projected/kube-api-access-vtkhz DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f79b5c8d-c736-4db0-93b4-b6a40ad58955/volumes/kubernetes.io~projected/kube-api-access-gxmwp DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/93b8a9b8-43df-4dc0-97da-4bfd11f373cd/volumes/kubernetes.io~projected/kube-api-access-66fd7 DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5/volumes/kubernetes.io~projected/kube-api-access-zt2sw DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd4258792ba0ee64ef0efd5c54ba63eb4d9a3086731777341a4373baea17c365/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c282411-f8cb-4c83-8959-52035df7cd24/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8ddcddb-5053-4950-bc4e-5d8f83ec8f08/volumes/kubernetes.io~projected/kube-api-access-qh7x7 DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94e9f3c857ebe0d4f5f59a923374a2f4f52bab39e81de3fa693780477ec79840/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ed9c43e8-604b-44f5-a20c-a3af86848c2c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be/volumes/kubernetes.io~projected/kube-api-access-f9pp4 DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7932d7cc-b9fe-423f-bc88-93711eaec908/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aafa2232431d7aa08a6ec916db80c0e6faef13bb2d75d93016de45d34f48c0db/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9c282411-f8cb-4c83-8959-52035df7cd24/volumes/kubernetes.io~projected/kube-api-access-cgr65 DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7e4f934-2451-4133-aa1a-ee1145ac2c41/volumes/kubernetes.io~projected/kube-api-access-64mgb DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36e6fc788aeb4842a93d5bf9647b4568b11f62808fd1e38e44f7afac45ef82f8/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6646d0f03b00475dcec78da3fdc2ce91bd173d2146c8e0553cfbfd8922b105f/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef582cf82ab8be0925f7cfafdf970d224a83c43f40197f79259d1143549e8226/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e259c703a3d326d9f791f89f8113b7d78f535765ec289b9b8484e171e9b7198/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6f9aa963-3cff-41d4-925c-79ba5db70d57/volumes/kubernetes.io~projected/kube-api-access-9xlkr DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e4c9fa87eb934f94cf97a58acb7e5b132e16fbf5bb86e6cae7cd11327af46f7/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1026393f-ba8e-408a-b7df-a1d03e047b1f/volumes/kubernetes.io~projected/kube-api-access-nwpb8 DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17/volumes/kubernetes.io~projected/kube-api-access-t929j DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ed9c43e8-604b-44f5-a20c-a3af86848c2c/volumes/kubernetes.io~projected/kube-api-access-ngkcw DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8538278266c801cf13a88f5be3d63f71f57679c2ccdf4d91f7f9eaa542bdcf8b/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9fd7b99fb365a96dc6069ed563255afadfd76487c90447756c14152518ff8bb/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1e1a5f5-7d62-43a9-9a51-1cb702e80963/volumes/kubernetes.io~projected/kube-api-access-n2zxc DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/303048ebdd7310eda3df6c6f626aa2573535edeb330043ab110100efe677f7cd/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5967c9990da73ecc128bfd5f29740b925371d8923b2b5015f791f1d88ef1905/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/21d827a7-6f18-4be7-86eb-b9141d12a44f/volumes/kubernetes.io~projected/kube-api-access-lnbxr DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/065bc8e2e2cc3204f157f053f695f5c1b35a88bd84faa84ae93555c2a70cbf9a/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49aa267f-cf95-4d9b-98c4-df106fcd9620/volumes/kubernetes.io~projected/kube-api-access-nf7x5 DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4dfa9a2f-f22b-4c85-a594-903928b2034a/volumes/kubernetes.io~projected/kube-api-access-dcgvd DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:065bc8e2e2cc320 MacAddress:de:b8:97:77:eb:6b Speed:10000 Mtu:8901} {Name:303048ebdd7310e MacAddress:82:eb:ae:66:13:66 Speed:10000 Mtu:8901} {Name:36e6fc788aeb484 MacAddress:22:fc:7d:2c:0a:10 Speed:10000 Mtu:8901} {Name:6ceea6bbf77a56a MacAddress:f2:a4:94:fe:6b:f4 Speed:10000 Mtu:8901} {Name:7b233a8a715e2f1 MacAddress:ba:7d:91:e3:d9:69 Speed:10000 Mtu:8901} {Name:a1174aee1f9265a MacAddress:d6:08:33:1c:e2:68 Speed:10000 Mtu:8901} {Name:aafa2232431d7aa MacAddress:7a:44:4e:c0:f7:2c Speed:10000 Mtu:8901} {Name:b8df4a6db68add4 MacAddress:1e:3e:0d:71:ad:13 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:dc:7a:96:71 Speed:0 Mtu:9001} {Name:br-int MacAddress:96:22:66:c9:ad:45 Speed:0 Mtu:8901} {Name:d9fd7b99fb365a9 MacAddress:32:36:9f:00:00:1a Speed:10000 Mtu:8901} {Name:dd4258792ba0ee6 MacAddress:9e:b6:b0:8a:e3:19 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:dc:7a:96:71 Speed:0 Mtu:9001} {Name:f5967c9990da73e MacAddress:a2:5a:d7:d4:25:c6 Speed:10000 Mtu:8901} {Name:ff86a527fde692c MacAddress:56:7a:82:b2:f9:cd Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:7a:c5:6b:83:4d:12 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:76:88:d0:35:79:de Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.748886 4861 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.748927 4861 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.749192 4861 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.749216 4861 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-179.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.749343 4861 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.749351 4861 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.749363 4861 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.749381 4861 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.749398 4861 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.749541 4861 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.749651 4861 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.749688 4861 kubelet.go:493] "Attempting to sync node with API server" Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.749702 4861 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.749712 4861 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.749750 4861 kubelet.go:405] "Adding apiserver pod source" Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.749758 4861 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.751934 4861 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 16:22:49.752688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.751949 4861 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:22:49.753469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.752758 4861 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 16:22:49.754773 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.754760 4861 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 16:22:49.754814 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.754773 4861 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:22:49.754873 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.754864 4861 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 16:22:49.755053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.755042 4861 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 16:22:49.755084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.755071 4861 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 16:22:49.755162 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.755154 4861 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 16:22:49.755195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.755168 4861 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 16:22:49.755195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.755173 4861 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 16:22:49.755195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.755178 4861 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 16:22:49.755195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.755183 4861 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 16:22:49.755195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.755189 4861 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 16:22:49.755195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.755194 4861 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 16:22:49.755362 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.755200 4861 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 16:22:49.755362 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.755207 4861 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 16:22:49.755362 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.755213 4861 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 16:22:49.755362 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.755221 4861 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 16:22:49.755362 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.755231 4861 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 16:22:49.755362 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.755256 4861 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 16:22:49.755362 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.755262 4861 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 16:22:49.758612 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.758601 4861 server.go:1263] "Started kubelet" Oct 30 16:22:49.758709 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.758680 4861 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 16:22:49.758787 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.758710 4861 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 16:22:49.758840 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.758822 4861 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 16:22:49.759203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.759183 4861 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 16:22:49.759207 ip-10-0-131-179 systemd[1]: Started Kubernetes Kubelet. Oct 30 16:22:49.761652 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.761635 4861 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 16:22:49.763467 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.762045 4861 server.go:317] "Adding debug handlers to kubelet server" Oct 30 16:22:49.763467 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:49.762130 4861 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-179.ec2.internal\" not found" Oct 30 16:22:49.763467 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.762670 4861 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 16:22:49.763467 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.762681 4861 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 16:22:49.763467 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.762804 4861 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 16:22:49.771196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.767650 4861 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 16:22:49.771196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.769281 4861 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 16:22:49.771196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.769294 4861 factory.go:55] Registering systemd factory Oct 30 16:22:49.771196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.769305 4861 factory.go:223] Registration of the systemd container factory successfully Oct 30 16:22:49.771196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.769377 4861 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:17:11 +0000 UTC" deadline="2028-06-21 17:32:14.095859034 +0000 UTC" Oct 30 16:22:49.771196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.769409 4861 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14401h9m24.326453326s" Oct 30 16:22:49.771196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.770788 4861 factory.go:153] Registering CRI-O factory Oct 30 16:22:49.771196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.770801 4861 factory.go:223] Registration of the crio container factory successfully Oct 30 16:22:49.771196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.770816 4861 factory.go:103] Registering Raw factory Oct 30 16:22:49.771196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.770826 4861 manager.go:1199] Started watching for new ooms in manager Oct 30 16:22:49.771196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.771095 4861 manager.go:319] Starting recovery of all containers Oct 30 16:22:49.773182 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773084 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7932d7cc-b9fe-423f-bc88-93711eaec908" volumeName="kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-kube-api-access-kghb4" seLinuxMountContext="" Oct 30 16:22:49.773285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773203 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c282411-f8cb-4c83-8959-52035df7cd24" volumeName="kubernetes.io/configmap/9c282411-f8cb-4c83-8959-52035df7cd24-service-ca-bundle" seLinuxMountContext="" Oct 30 16:22:49.773285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773220 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4922dc9-bd15-45a0-a9f6-e61089e6096b" volumeName="kubernetes.io/configmap/a4922dc9-bd15-45a0-a9f6-e61089e6096b-config" seLinuxMountContext="" Oct 30 16:22:49.773285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773249 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed9c43e8-604b-44f5-a20c-a3af86848c2c" volumeName="kubernetes.io/projected/ed9c43e8-604b-44f5-a20c-a3af86848c2c-kube-api-access-ngkcw" seLinuxMountContext="" Oct 30 16:22:49.773285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773261 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6089204f-1338-4875-b311-2fa1b27a03d5" volumeName="kubernetes.io/configmap/6089204f-1338-4875-b311-2fa1b27a03d5-konnectivity-ca" seLinuxMountContext="" Oct 30 16:22:49.773285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773273 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7932d7cc-b9fe-423f-bc88-93711eaec908" volumeName="kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-installation-pull-secrets" seLinuxMountContext="" Oct 30 16:22:49.773542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773288 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f" volumeName="kubernetes.io/configmap/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-iptables-alerter-script" seLinuxMountContext="" Oct 30 16:22:49.773542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773310 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8ddcddb-5053-4950-bc4e-5d8f83ec8f08" volumeName="kubernetes.io/projected/c8ddcddb-5053-4950-bc4e-5d8f83ec8f08-kube-api-access-qh7x7" seLinuxMountContext="" Oct 30 16:22:49.773542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773355 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb9bd726-ba59-4c55-838f-e00276ca9bef" volumeName="kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret" seLinuxMountContext="" Oct 30 16:22:49.773542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773388 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58c9bbd6-243d-4769-afc9-19e5e4097ef9" volumeName="kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cni-binary-copy" seLinuxMountContext="" Oct 30 16:22:49.773542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773406 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7932d7cc-b9fe-423f-bc88-93711eaec908" volumeName="kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-trusted-ca" seLinuxMountContext="" Oct 30 16:22:49.773542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773439 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ab0447a-bdef-45aa-92c3-e9d35ee9c2be" volumeName="kubernetes.io/projected/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-kube-api-access-f9pp4" seLinuxMountContext="" Oct 30 16:22:49.773542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773452 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d0df1a2-2140-46d5-8662-1148296fb5a8" volumeName="kubernetes.io/empty-dir/1d0df1a2-2140-46d5-8662-1148296fb5a8-tmp-dir" seLinuxMountContext="" Oct 30 16:22:49.773542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773478 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7932d7cc-b9fe-423f-bc88-93711eaec908" volumeName="kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-image-registry-private-configuration" seLinuxMountContext="" Oct 30 16:22:49.773542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773508 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fff2d07-9f9e-4edb-a32c-ad3a9440fc17" volumeName="kubernetes.io/projected/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-kube-api-access-t929j" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773537 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79b5c8d-c736-4db0-93b4-b6a40ad58955" volumeName="kubernetes.io/configmap/f79b5c8d-c736-4db0-93b4-b6a40ad58955-config-volume" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773559 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79b5c8d-c736-4db0-93b4-b6a40ad58955" volumeName="kubernetes.io/empty-dir/f79b5c8d-c736-4db0-93b4-b6a40ad58955-tmp-dir" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773589 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49aa267f-cf95-4d9b-98c4-df106fcd9620" volumeName="kubernetes.io/projected/49aa267f-cf95-4d9b-98c4-df106fcd9620-kube-api-access-nf7x5" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773601 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c282411-f8cb-4c83-8959-52035df7cd24" volumeName="kubernetes.io/configmap/9c282411-f8cb-4c83-8959-52035df7cd24-trusted-ca-bundle" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773623 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4922dc9-bd15-45a0-a9f6-e61089e6096b" volumeName="kubernetes.io/projected/a4922dc9-bd15-45a0-a9f6-e61089e6096b-kube-api-access-ldgcm" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773639 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed9c43e8-604b-44f5-a20c-a3af86848c2c" volumeName="kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovnkube-script-lib" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773663 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ab95e94-bdff-4f1f-a0fe-095a2509d6f8" volumeName="kubernetes.io/projected/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-kube-api-access-4qglw" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773680 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7932d7cc-b9fe-423f-bc88-93711eaec908" volumeName="kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-bound-sa-token" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773692 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c282411-f8cb-4c83-8959-52035df7cd24" volumeName="kubernetes.io/empty-dir/9c282411-f8cb-4c83-8959-52035df7cd24-tmp" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773703 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f79b5c8d-c736-4db0-93b4-b6a40ad58955" volumeName="kubernetes.io/projected/f79b5c8d-c736-4db0-93b4-b6a40ad58955-kube-api-access-gxmwp" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773758 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dfa9a2f-f22b-4c85-a594-903928b2034a" volumeName="kubernetes.io/projected/4dfa9a2f-f22b-4c85-a594-903928b2034a-kube-api-access-dcgvd" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773771 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58c9bbd6-243d-4769-afc9-19e5e4097ef9" volumeName="kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773798 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed9c43e8-604b-44f5-a20c-a3af86848c2c" volumeName="kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-env-overrides" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773843 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" volumeName="kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-certificates" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773856 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" volumeName="kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-trusted-ca" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773867 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" volumeName="kubernetes.io/empty-dir/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-ca-trust-extracted" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773894 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12874a88-98a6-436e-973d-cb931fc769de" volumeName="kubernetes.io/projected/12874a88-98a6-436e-973d-cb931fc769de-kube-api-access-8q6vt" seLinuxMountContext="" Oct 30 16:22:49.773983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773918 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ab0447a-bdef-45aa-92c3-e9d35ee9c2be" volumeName="kubernetes.io/configmap/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-serviceca" seLinuxMountContext="" Oct 30 16:22:49.774673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.773982 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a21d3561-30d9-4a48-816b-5b7be4e5813d" volumeName="kubernetes.io/projected/a21d3561-30d9-4a48-816b-5b7be4e5813d-kube-api-access-lqhlw" seLinuxMountContext="" Oct 30 16:22:49.774673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.774009 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f9aa963-3cff-41d4-925c-79ba5db70d57" volumeName="kubernetes.io/configmap/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-daemon-config" seLinuxMountContext="" Oct 30 16:22:49.774673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.774024 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7932d7cc-b9fe-423f-bc88-93711eaec908" volumeName="kubernetes.io/empty-dir/7932d7cc-b9fe-423f-bc88-93711eaec908-ca-trust-extracted" seLinuxMountContext="" Oct 30 16:22:49.774673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.774041 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fff2d07-9f9e-4edb-a32c-ad3a9440fc17" volumeName="kubernetes.io/configmap/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-config" seLinuxMountContext="" Oct 30 16:22:49.774673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.774087 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b8a9b8-43df-4dc0-97da-4bfd11f373cd" volumeName="kubernetes.io/configmap/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-telemetry-config" seLinuxMountContext="" Oct 30 16:22:49.774673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.774106 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ab95e94-bdff-4f1f-a0fe-095a2509d6f8" volumeName="kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-stats-auth" seLinuxMountContext="" Oct 30 16:22:49.774673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.774150 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f4d6719-619b-46dd-9b1d-93fd5d1e91c0" volumeName="kubernetes.io/projected/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-kube-api-access-lsjwr" seLinuxMountContext="" Oct 30 16:22:49.774673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.774164 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a25e3b8-4f22-4fdb-86a9-70d1c594236d" volumeName="kubernetes.io/secret/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-serving-cert" seLinuxMountContext="" Oct 30 16:22:49.774673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.774190 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7932d7cc-b9fe-423f-bc88-93711eaec908" volumeName="kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-certificates" seLinuxMountContext="" Oct 30 16:22:49.774673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.774232 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed9c43e8-604b-44f5-a20c-a3af86848c2c" volumeName="kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovnkube-config" seLinuxMountContext="" Oct 30 16:22:49.774673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.774249 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed9c43e8-604b-44f5-a20c-a3af86848c2c" volumeName="kubernetes.io/secret/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 16:22:49.774673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.774264 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1026393f-ba8e-408a-b7df-a1d03e047b1f" volumeName="kubernetes.io/projected/1026393f-ba8e-408a-b7df-a1d03e047b1f-kube-api-access-nwpb8" seLinuxMountContext="" Oct 30 16:22:49.774673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.774283 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21d827a7-6f18-4be7-86eb-b9141d12a44f" volumeName="kubernetes.io/empty-dir/21d827a7-6f18-4be7-86eb-b9141d12a44f-data-volume" seLinuxMountContext="" Oct 30 16:22:49.775549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775516 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4dfa9a2f-f22b-4c85-a594-903928b2034a" volumeName="kubernetes.io/secret/4dfa9a2f-f22b-4c85-a594-903928b2034a-cert" seLinuxMountContext="" Oct 30 16:22:49.775642 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775560 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58c9bbd6-243d-4769-afc9-19e5e4097ef9" volumeName="kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 16:22:49.775642 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775577 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cff25808-bc4e-4bd3-b38c-5503a00d45fa" volumeName="kubernetes.io/projected/cff25808-bc4e-4bd3-b38c-5503a00d45fa-kube-api-access-rt58k" seLinuxMountContext="" Oct 30 16:22:49.775642 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775591 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddae22b7-ddc1-4766-9af6-3899f1edf48c" volumeName="kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc" seLinuxMountContext="" Oct 30 16:22:49.775642 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775606 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" volumeName="kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-image-registry-private-configuration" seLinuxMountContext="" Oct 30 16:22:49.775642 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775619 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" volumeName="kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-installation-pull-secrets" seLinuxMountContext="" Oct 30 16:22:49.775642 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775632 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6089204f-1338-4875-b311-2fa1b27a03d5" volumeName="kubernetes.io/secret/6089204f-1338-4875-b311-2fa1b27a03d5-agent-certs" seLinuxMountContext="" Oct 30 16:22:49.775855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775644 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1e1a5f5-7d62-43a9-9a51-1cb702e80963" volumeName="kubernetes.io/configmap/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-config" seLinuxMountContext="" Oct 30 16:22:49.775855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775658 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1e1a5f5-7d62-43a9-9a51-1cb702e80963" volumeName="kubernetes.io/projected/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-kube-api-access-n2zxc" seLinuxMountContext="" Oct 30 16:22:49.775855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775700 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4922dc9-bd15-45a0-a9f6-e61089e6096b" volumeName="kubernetes.io/secret/a4922dc9-bd15-45a0-a9f6-e61089e6096b-serving-cert" seLinuxMountContext="" Oct 30 16:22:49.775855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775750 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7e4f934-2451-4133-aa1a-ee1145ac2c41" volumeName="kubernetes.io/projected/a7e4f934-2451-4133-aa1a-ee1145ac2c41-kube-api-access-64mgb" seLinuxMountContext="" Oct 30 16:22:49.775855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775765 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" volumeName="kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-bound-sa-token" seLinuxMountContext="" Oct 30 16:22:49.775855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775782 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" volumeName="kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-kube-api-access-zt2sw" seLinuxMountContext="" Oct 30 16:22:49.775855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775800 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21d827a7-6f18-4be7-86eb-b9141d12a44f" volumeName="kubernetes.io/projected/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-api-access-lnbxr" seLinuxMountContext="" Oct 30 16:22:49.775855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775813 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a25e3b8-4f22-4fdb-86a9-70d1c594236d" volumeName="kubernetes.io/projected/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-kube-api-access-fc6nb" seLinuxMountContext="" Oct 30 16:22:49.775855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775828 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93b8a9b8-43df-4dc0-97da-4bfd11f373cd" volumeName="kubernetes.io/projected/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-kube-api-access-66fd7" seLinuxMountContext="" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775858 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c282411-f8cb-4c83-8959-52035df7cd24" volumeName="kubernetes.io/projected/9c282411-f8cb-4c83-8959-52035df7cd24-kube-api-access-cgr65" seLinuxMountContext="" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775873 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f4d6719-619b-46dd-9b1d-93fd5d1e91c0" volumeName="kubernetes.io/empty-dir/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-tmp" seLinuxMountContext="" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775886 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58c9bbd6-243d-4769-afc9-19e5e4097ef9" volumeName="kubernetes.io/projected/58c9bbd6-243d-4769-afc9-19e5e4097ef9-kube-api-access-tttc4" seLinuxMountContext="" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775901 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c282411-f8cb-4c83-8959-52035df7cd24" volumeName="kubernetes.io/secret/9c282411-f8cb-4c83-8959-52035df7cd24-serving-cert" seLinuxMountContext="" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775915 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1ab95e94-bdff-4f1f-a0fe-095a2509d6f8" volumeName="kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-default-certificate" seLinuxMountContext="" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775930 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c282411-f8cb-4c83-8959-52035df7cd24" volumeName="kubernetes.io/empty-dir/9c282411-f8cb-4c83-8959-52035df7cd24-snapshots" seLinuxMountContext="" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775942 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f" volumeName="kubernetes.io/projected/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-kube-api-access-vtkhz" seLinuxMountContext="" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775956 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d0df1a2-2140-46d5-8662-1148296fb5a8" volumeName="kubernetes.io/projected/1d0df1a2-2140-46d5-8662-1148296fb5a8-kube-api-access-gzxpj" seLinuxMountContext="" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775968 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a25e3b8-4f22-4fdb-86a9-70d1c594236d" volumeName="kubernetes.io/configmap/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-trusted-ca" seLinuxMountContext="" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775984 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f9aa963-3cff-41d4-925c-79ba5db70d57" volumeName="kubernetes.io/configmap/6f9aa963-3cff-41d4-925c-79ba5db70d57-cni-binary-copy" seLinuxMountContext="" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.775998 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f9aa963-3cff-41d4-925c-79ba5db70d57" volumeName="kubernetes.io/projected/6f9aa963-3cff-41d4-925c-79ba5db70d57-kube-api-access-9xlkr" seLinuxMountContext="" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.776010 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1e1a5f5-7d62-43a9-9a51-1cb702e80963" volumeName="kubernetes.io/secret/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-serving-cert" seLinuxMountContext="" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.776023 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f4d6719-619b-46dd-9b1d-93fd5d1e91c0" volumeName="kubernetes.io/empty-dir/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-tuned" seLinuxMountContext="" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.776036 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a25e3b8-4f22-4fdb-86a9-70d1c594236d" volumeName="kubernetes.io/configmap/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-config" seLinuxMountContext="" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.776047 4861 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78ae398a-fdb1-48f8-94fa-51eb582e663a" volumeName="kubernetes.io/configmap/78ae398a-fdb1-48f8-94fa-51eb582e663a-nginx-conf" seLinuxMountContext="" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.776059 4861 reconstruct.go:97] "Volume reconstruction finished" Oct 30 16:22:49.776112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.776067 4861 reconciler.go:29] "Reconciler: start to sync state" Oct 30 16:22:49.786148 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.786115 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/1.log" Oct 30 16:22:49.787060 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.787040 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/1.log" Oct 30 16:22:49.797469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.797441 4861 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 16:22:49.798933 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.798906 4861 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 16:22:49.798933 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.798936 4861 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 16:22:49.799070 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.798956 4861 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 16:22:49.799070 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:49.798995 4861 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 16:22:49.815226 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.815200 4861 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:49.815328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.815270 4861 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 16:22:49.815592 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.815573 4861 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:49.815690 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.815598 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/1.log" Oct 30 16:22:49.815690 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.815626 4861 generic.go:356] "Generic (PLEG): container finished" podID="5a25e3b8-4f22-4fdb-86a9-70d1c594236d" containerID="9327de775d1ac0e11a7137ae4ef9fb291b5a69127a8688d920be8a7a4ccfe5fa" exitCode=255 Oct 30 16:22:49.827652 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.827626 4861 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:49.828011 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.827983 4861 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:22:49.834457 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.834433 4861 generic.go:356] "Generic (PLEG): container finished" podID="58c9bbd6-243d-4769-afc9-19e5e4097ef9" containerID="05ba1822c46a34496edf469090f4762df1503e228f85580a80e0a8af1bd42cfa" exitCode=0 Oct 30 16:22:49.834457 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.834456 4861 generic.go:356] "Generic (PLEG): container finished" podID="58c9bbd6-243d-4769-afc9-19e5e4097ef9" containerID="104331adf15444592c34876d6dd4e0a0e7f579796eb5b34de0fcbcfd0ad959e4" exitCode=0 Oct 30 16:22:49.834547 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.834462 4861 generic.go:356] "Generic (PLEG): container finished" podID="58c9bbd6-243d-4769-afc9-19e5e4097ef9" containerID="a526b901f5a420caed94ddb69934b7708dc148cdc4f21e92c388af2fc13f8e3c" exitCode=0 Oct 30 16:22:49.834547 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.834469 4861 generic.go:356] "Generic (PLEG): container finished" podID="58c9bbd6-243d-4769-afc9-19e5e4097ef9" containerID="92271daeb0ef2812da5c46ab810a578bb74fa352a8bd7f18723595b249bdcc6a" exitCode=0 Oct 30 16:22:49.834547 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.834475 4861 generic.go:356] "Generic (PLEG): container finished" podID="58c9bbd6-243d-4769-afc9-19e5e4097ef9" containerID="cefcace71706c7a063c687137b0ead8c2a7e78eed98ae2040cd6ee95d4d35dfd" exitCode=0 Oct 30 16:22:49.834547 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.834480 4861 generic.go:356] "Generic (PLEG): container finished" podID="58c9bbd6-243d-4769-afc9-19e5e4097ef9" containerID="432586f6b7be6ff757a5bdb7a88159ef8000b9cdf15c5370a62676be257ddfa1" exitCode=0 Oct 30 16:22:49.837889 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.837868 4861 generic.go:356] "Generic (PLEG): container finished" podID="26fcf7a73a4a25dbdf26c9ac58f465b2" containerID="d117de0a8667e68e2ce3cb475691f5c74168e308d5b5fb15d6658378709cfea6" exitCode=0 Oct 30 16:22:49.861332 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.861285 4861 manager.go:324] Recovery completed Oct 30 16:22:49.879598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.879576 4861 cpu_manager.go:226] "Starting" policy="none" Oct 30 16:22:49.879598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.879588 4861 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 16:22:49.879743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.879603 4861 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 16:22:49.879743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.879706 4861 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 16:22:49.879743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.879715 4861 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 16:22:49.879874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.879798 4861 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 16:22:49.879874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.879804 4861 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 16:22:49.879874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.879809 4861 policy_none.go:50] "Start" Oct 30 16:22:49.879874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.879817 4861 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 16:22:49.879874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.879826 4861 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 16:22:49.880007 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.879944 4861 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 16:22:49.880007 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.879951 4861 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 16:22:49.880007 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.879956 4861 policy_none.go:44] "Start" Oct 30 16:22:49.884193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.884170 4861 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 16:22:49.886948 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.886919 4861 manager.go:341] "Starting Device Plugin manager" Oct 30 16:22:49.888915 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:49.886969 4861 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 16:22:49.889033 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.888920 4861 server.go:87] "Starting device plugin registration server" Oct 30 16:22:49.889296 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.889246 4861 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 16:22:49.889296 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.889263 4861 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 16:22:49.889655 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.889628 4861 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 16:22:49.889781 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.889745 4861 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 16:22:49.889857 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.889788 4861 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 16:22:49.889857 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.889795 4861 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 16:22:49.890076 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.889844 4861 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:22:49.889778328Z","UUID":"98afa8f5-491a-4c86-8b3d-b535349a4259","Handler":null,"Name":"","Endpoint":""} Oct 30 16:22:49.891504 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:49.891479 4861 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 16:22:49.891578 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.891488 4861 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 16:22:49.891578 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.891555 4861 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 16:22:49.899109 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.899059 4861 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal"] Oct 30 16:22:49.899709 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.899672 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" event={"ID":"26f3b181327243d25a7eb8229eca8f4d","Type":"ContainerStarted","Data":"fcdf89b54a25e07b2dc3e353b69e8df060d3e81bbe6cd9ab0200b53caa9b1e41"} Oct 30 16:22:49.899820 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.899710 4861 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:49.899820 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.899770 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" event={"ID":"26f3b181327243d25a7eb8229eca8f4d","Type":"ContainerStarted","Data":"8e259c703a3d326d9f791f89f8113b7d78f535765ec289b9b8484e171e9b7198"} Oct 30 16:22:49.899820 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.899820 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" event={"ID":"26fcf7a73a4a25dbdf26c9ac58f465b2","Type":"ContainerStarted","Data":"d9aa510e315096213569a9948c9ae1392c6783b383abd47232d1b7d5ade8361f"} Oct 30 16:22:49.899963 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.899834 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" event={"ID":"26fcf7a73a4a25dbdf26c9ac58f465b2","Type":"ContainerDied","Data":"d117de0a8667e68e2ce3cb475691f5c74168e308d5b5fb15d6658378709cfea6"} Oct 30 16:22:49.899963 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.899846 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" event={"ID":"26fcf7a73a4a25dbdf26c9ac58f465b2","Type":"ContainerStarted","Data":"8538278266c801cf13a88f5be3d63f71f57679c2ccdf4d91f7f9eaa542bdcf8b"} Oct 30 16:22:49.900028 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.899979 4861 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Oct 30 16:22:49.924640 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.924624 4861 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:22:49.924742 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:49.924663 4861 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-179.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Oct 30 16:22:49.925129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.925113 4861 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:22:49.925173 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:49.925150 4861 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:49.989388 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.989374 4861 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:22:49.990644 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.990631 4861 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:22:49.990704 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.990656 4861 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:22:49.990704 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.990665 4861 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-179.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:22:49.990704 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:49.990704 4861 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-179.ec2.internal" Oct 30 16:22:50.015911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.015893 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fcf7a73a4a25dbdf26c9ac58f465b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"26fcf7a73a4a25dbdf26c9ac58f465b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:50.015996 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.015919 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/26f3b181327243d25a7eb8229eca8f4d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-179.ec2.internal\" (UID: \"26f3b181327243d25a7eb8229eca8f4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Oct 30 16:22:50.015996 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.015935 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26fcf7a73a4a25dbdf26c9ac58f465b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"26fcf7a73a4a25dbdf26c9ac58f465b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:50.027039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.027022 4861 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-179.ec2.internal" Oct 30 16:22:50.027078 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.027070 4861 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-179.ec2.internal" Oct 30 16:22:50.116175 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.116125 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26fcf7a73a4a25dbdf26c9ac58f465b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"26fcf7a73a4a25dbdf26c9ac58f465b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:50.116175 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.116150 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fcf7a73a4a25dbdf26c9ac58f465b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"26fcf7a73a4a25dbdf26c9ac58f465b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:50.116275 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.116195 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26fcf7a73a4a25dbdf26c9ac58f465b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"26fcf7a73a4a25dbdf26c9ac58f465b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:50.116275 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.116230 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/26f3b181327243d25a7eb8229eca8f4d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-179.ec2.internal\" (UID: \"26f3b181327243d25a7eb8229eca8f4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Oct 30 16:22:50.116275 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.116230 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26fcf7a73a4a25dbdf26c9ac58f465b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" (UID: \"26fcf7a73a4a25dbdf26c9ac58f465b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:50.116356 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.116281 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/26f3b181327243d25a7eb8229eca8f4d-config\") pod \"kube-apiserver-proxy-ip-10-0-131-179.ec2.internal\" (UID: \"26f3b181327243d25a7eb8229eca8f4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Oct 30 16:22:50.750427 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.750405 4861 apiserver.go:52] "Watching apiserver" Oct 30 16:22:50.792460 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.792433 4861 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 16:22:50.794318 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.794285 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-kx55q","kube-system/global-pull-secret-syncer-pq64k","openshift-network-console/networking-console-plugin-86c478bb88-6xx9g","openshift-ingress/router-default-58c95ddf47-6m2jw","kube-system/konnectivity-agent-gjhv5","kube-system/kas-connection-checker-685895c7c9-9b9bn","openshift-cluster-node-tuning-operator/tuned-rwx47","openshift-ingress-canary/ingress-canary-mfwcr","openshift-network-diagnostics/network-check-source-6f78d59f99-sj6h8","openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm","openshift-network-diagnostics/network-check-target-hptcz","openshift-multus/multus-wr6tc","openshift-multus/network-metrics-daemon-fz7kn","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44","openshift-network-operator/iptables-alerter-265mk","openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp","kube-system/kas-connection-checker-685895c7c9-5r74t","openshift-image-registry/node-ca-pbsvn","openshift-console-operator/console-operator-7cd56b8dbc-szx5s","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7","openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8","openshift-ovn-kubernetes/ovnkube-node-l95sm","openshift-dns/node-resolver-6xn9h","openshift-insights/insights-runtime-extractor-wnmqt","kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal","openshift-dns/dns-default-82w6j","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf","openshift-multus/multus-additional-cni-plugins-7stpt","kube-system/kas-connection-checker-685895c7c9-2sc7w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal","openshift-image-registry/image-registry-5dfdc4b7f-lkk6g","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rmgmf"] Oct 30 16:22:50.796262 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.796236 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:50.796649 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.796632 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:50.797075 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.797045 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:50.798401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.798385 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:50.799225 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.799209 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:50.802990 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.802970 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:50.803542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.803527 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:50.807035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.807019 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:50.807911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.807894 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:50.809834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.809820 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.810007 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.809992 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.810089 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.810035 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.811084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.811070 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 16:22:50.811176 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.811163 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 16:22:50.811225 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.811193 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.815242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.813021 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xdq5p\"" Oct 30 16:22:50.815242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.813157 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.815242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.813302 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.815242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.813800 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.815242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.813844 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 16:22:50.815242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.813865 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 16:22:50.815242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.813937 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-wkbtq\"" Oct 30 16:22:50.815242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.814009 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r5btp\"" Oct 30 16:22:50.815242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.814051 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 16:22:50.815242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.814150 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 16:22:50.818829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.818806 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 16:22:50.819033 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.819018 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 16:22:50.819290 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.819278 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.821952 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.821935 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 16:22:50.822046 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.822032 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lwtv6\"" Oct 30 16:22:50.823777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.823767 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-86hj4\"" Oct 30 16:22:50.823877 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.823782 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 16:22:50.823877 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.823843 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 16:22:50.823877 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.823845 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.824398 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824381 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.824490 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824396 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 16:22:50.824490 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824408 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.824490 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824438 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 16:22:50.824633 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824533 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 16:22:50.824633 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824545 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.824633 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824604 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 16:22:50.824633 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824628 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.824825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824628 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.824825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824664 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.824825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824680 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.824825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824700 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 16:22:50.824825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824699 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 16:22:50.824825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824714 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 16:22:50.824825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824762 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.824825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824761 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.824825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824778 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 16:22:50.824825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824764 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.824825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824790 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.824825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824803 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.824825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824818 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.824825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824820 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.825330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824958 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 16:22:50.825330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.824995 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 16:22:50.825330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.825012 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wm4jw\"" Oct 30 16:22:50.825330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.825084 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 16:22:50.825330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.825101 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-m9x9f\"" Oct 30 16:22:50.825330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.825151 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 16:22:50.825330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.825206 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6n6gp\"" Oct 30 16:22:50.829207 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.829194 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.829451 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.829434 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h4vt2\"" Oct 30 16:22:50.829558 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.829542 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 16:22:50.829624 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.829543 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 16:22:50.829624 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.829552 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.830843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.830823 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.830960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.830856 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 16:22:50.830960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.830857 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 16:22:50.830960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.830875 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 16:22:50.830960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.830856 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 16:22:50.830960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.830941 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 16:22:50.831364 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.831348 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 16:22:50.831364 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.831357 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:22:50.831474 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.831391 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.831474 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.831428 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.831474 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.831428 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.831601 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.831476 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.831779 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.831765 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.831837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.831781 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.831837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.831812 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.831943 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.831858 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 16:22:50.831943 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.831882 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lf54z\"" Oct 30 16:22:50.831943 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.831888 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bd59p\"" Oct 30 16:22:50.831943 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.831924 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cn6jt\"" Oct 30 16:22:50.832124 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832085 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pzcq9\"" Oct 30 16:22:50.832177 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832128 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 16:22:50.832177 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832133 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d6852\"" Oct 30 16:22:50.832177 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832172 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.832311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832173 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tdd2t\"" Oct 30 16:22:50.832311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832190 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vhqb4\"" Oct 30 16:22:50.832311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832209 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 16:22:50.832311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832238 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rjhbz\"" Oct 30 16:22:50.832311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832253 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hpqxb\"" Oct 30 16:22:50.832311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832285 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 16:22:50.832311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832294 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5n7vr\"" Oct 30 16:22:50.832615 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832318 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-65hn8\"" Oct 30 16:22:50.832615 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832367 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 16:22:50.832615 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832384 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 16:22:50.832615 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832394 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 16:22:50.832615 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832403 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 16:22:50.832615 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832395 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 16:22:50.832911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832672 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-nnwcz\"" Oct 30 16:22:50.832911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832816 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:22:50.832911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832841 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-vqbh2\"" Oct 30 16:22:50.832911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832882 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9txpl\"" Oct 30 16:22:50.832911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832894 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lpvkt\"" Oct 30 16:22:50.832911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832903 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z9zpr\"" Oct 30 16:22:50.833079 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.832939 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 16:22:50.833178 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.833167 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9kf5z\"" Oct 30 16:22:50.836404 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.836390 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 16:22:50.837067 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.837054 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 16:22:50.837183 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.837170 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 16:22:50.851560 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.851545 4861 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Oct 30 16:22:50.851731 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.851709 4861 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:50.865292 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.865274 4861 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 16:22:50.873355 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.873340 4861 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:22:50.873399 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:50.873377 4861 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-179.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-179.ec2.internal" Oct 30 16:22:50.873750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.873738 4861 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:22:50.873785 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:50.873772 4861 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-179.ec2.internal" Oct 30 16:22:50.921606 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.921582 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.921751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.921608 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:50.921751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.921632 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-config\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:50.921882 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.921754 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysconfig\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.921882 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.921788 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-lib-modules\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.921882 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.921815 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-node-log\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.921882 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.921864 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-system-cni-dir\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:50.922039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.921897 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-etc-kubernetes\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.922039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.921922 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2zxc\" (UniqueName: \"kubernetes.io/projected/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-kube-api-access-n2zxc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wgr44\" (UID: \"a1e1a5f5-7d62-43a9-9a51-1cb702e80963\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:50.922039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.921924 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-config\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:50.922039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.921941 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-modprobe-d\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.922039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.921957 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-tmp\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.922039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.921979 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-etc-selinux\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:50.922039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.921996 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cni-binary-copy\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:50.922039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.922011 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-installation-pull-secrets\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:50.922039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.922027 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d0df1a2-2140-46d5-8662-1148296fb5a8-tmp-dir\") pod \"node-resolver-6xn9h\" (UID: \"1d0df1a2-2140-46d5-8662-1148296fb5a8\") " pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:50.922357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.922049 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-tmp\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.922357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.922093 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d0df1a2-2140-46d5-8662-1148296fb5a8-tmp-dir\") pod \"node-resolver-6xn9h\" (UID: \"1d0df1a2-2140-46d5-8662-1148296fb5a8\") " pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:50.922357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.922128 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c282411-f8cb-4c83-8959-52035df7cd24-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:50.922357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.922152 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-socket-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:50.922357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.922170 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kghb4\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-kube-api-access-kghb4\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:50.922357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.922185 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-cni-multus\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.922623 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.922525 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-installation-pull-secrets\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:50.922623 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.922564 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-default-certificate\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:50.922707 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.922618 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.922949 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.922923 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-default-certificate\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:50.923034 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.922984 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c282411-f8cb-4c83-8959-52035df7cd24-tmp\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:50.923034 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.922986 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c282411-f8cb-4c83-8959-52035df7cd24-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:50.923034 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.923018 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-serviceca\") pod \"node-ca-pbsvn\" (UID: \"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be\") " pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:50.923222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.923056 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:50.923222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.923090 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-var-lib-kubelet\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.923222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.923097 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cni-binary-copy\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:50.923222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.923116 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.923222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.923153 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-sys-fs\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:50.923222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.923192 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-bound-sa-token\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:50.923546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.923243 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:50.923546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.923281 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzxpj\" (UniqueName: \"kubernetes.io/projected/1d0df1a2-2140-46d5-8662-1148296fb5a8-kube-api-access-gzxpj\") pod \"node-resolver-6xn9h\" (UID: \"1d0df1a2-2140-46d5-8662-1148296fb5a8\") " pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:50.923546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.923382 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c282411-f8cb-4c83-8959-52035df7cd24-tmp\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:50.923546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.923383 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-serviceca\") pod \"node-ca-pbsvn\" (UID: \"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be\") " pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:50.923546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.923530 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqhlw\" (UniqueName: \"kubernetes.io/projected/a21d3561-30d9-4a48-816b-5b7be4e5813d-kube-api-access-lqhlw\") pod \"volume-data-source-validator-58c9f5f46d-rmgmf\" (UID: \"a21d3561-30d9-4a48-816b-5b7be4e5813d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rmgmf" Oct 30 16:22:50.923798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.923605 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cb9bd726-ba59-4c55-838f-e00276ca9bef-original-pull-secret\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:50.926221 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.926200 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:50.926319 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.926251 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64mgb\" (UniqueName: \"kubernetes.io/projected/a7e4f934-2451-4133-aa1a-ee1145ac2c41-kube-api-access-64mgb\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:50.926374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.926321 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21d827a7-6f18-4be7-86eb-b9141d12a44f-crio-socket\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:50.926374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.926365 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:50.926455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.926396 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c282411-f8cb-4c83-8959-52035df7cd24-service-ca-bundle\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:50.926496 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.926429 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:50.926573 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.926555 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwpb8\" (UniqueName: \"kubernetes.io/projected/1026393f-ba8e-408a-b7df-a1d03e047b1f-kube-api-access-nwpb8\") pod \"kas-connection-checker-685895c7c9-5r74t\" (UID: \"1026393f-ba8e-408a-b7df-a1d03e047b1f\") " pod="kube-system/kas-connection-checker-685895c7c9-5r74t" Oct 30 16:22:50.926691 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.926624 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c282411-f8cb-4c83-8959-52035df7cd24-service-ca-bundle\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:50.927014 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.926991 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4922dc9-bd15-45a0-a9f6-e61089e6096b-serving-cert\") pod \"service-ca-operator-544d749c8c-p79fm\" (UID: \"a4922dc9-bd15-45a0-a9f6-e61089e6096b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:50.927121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927024 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb9bd726-ba59-4c55-838f-e00276ca9bef-dbus\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:50.927121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927044 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6xx9g\" (UID: \"78ae398a-fdb1-48f8-94fa-51eb582e663a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:50.927121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927060 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-host\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.927121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927065 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:50.927121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927077 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-tuned\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.927121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927098 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-netns\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.927393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927144 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xlkr\" (UniqueName: \"kubernetes.io/projected/6f9aa963-3cff-41d4-925c-79ba5db70d57-kube-api-access-9xlkr\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.927393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927185 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4922dc9-bd15-45a0-a9f6-e61089e6096b-serving-cert\") pod \"service-ca-operator-544d749c8c-p79fm\" (UID: \"a4922dc9-bd15-45a0-a9f6-e61089e6096b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:50.927393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927205 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9c282411-f8cb-4c83-8959-52035df7cd24-snapshots\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:50.927393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927219 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-tuned\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.927393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927296 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f79b5c8d-c736-4db0-93b4-b6a40ad58955-tmp-dir\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:50.927393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927321 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9c282411-f8cb-4c83-8959-52035df7cd24-snapshots\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:50.927393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927360 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-var-lib-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.927593 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927413 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7932d7cc-b9fe-423f-bc88-93711eaec908-ca-trust-extracted\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:50.927593 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927430 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f79b5c8d-c736-4db0-93b4-b6a40ad58955-tmp-dir\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:50.927593 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927445 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:50.927593 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927479 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgr65\" (UniqueName: \"kubernetes.io/projected/9c282411-f8cb-4c83-8959-52035df7cd24-kube-api-access-cgr65\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:50.927593 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927502 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-multus-certs\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.927593 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927523 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-systemd\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.927593 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927526 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7932d7cc-b9fe-423f-bc88-93711eaec908-ca-trust-extracted\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:50.927593 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927544 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6089204f-1338-4875-b311-2fa1b27a03d5-agent-certs\") pod \"konnectivity-agent-gjhv5\" (UID: \"6089204f-1338-4875-b311-2fa1b27a03d5\") " pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:22:50.927593 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927579 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qglw\" (UniqueName: \"kubernetes.io/projected/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-kube-api-access-4qglw\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:50.927991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927641 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysctl-conf\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.927991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927713 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-run-netns\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.927991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927823 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-log-socket\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.927991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927844 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-env-overrides\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.927991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927859 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:50.927991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927877 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6089204f-1338-4875-b311-2fa1b27a03d5-agent-certs\") pod \"konnectivity-agent-gjhv5\" (UID: \"6089204f-1338-4875-b311-2fa1b27a03d5\") " pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:22:50.927991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927879 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldgcm\" (UniqueName: \"kubernetes.io/projected/a4922dc9-bd15-45a0-a9f6-e61089e6096b-kube-api-access-ldgcm\") pod \"service-ca-operator-544d749c8c-p79fm\" (UID: \"a4922dc9-bd15-45a0-a9f6-e61089e6096b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:50.927991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.927932 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f9aa963-3cff-41d4-925c-79ba5db70d57-cni-binary-copy\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.928327 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928051 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-kubelet\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.928327 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928080 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-k8s-cni-cncf-io\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.928327 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928102 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-slash\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.928327 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928135 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-env-overrides\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.928327 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928147 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58c9bbd6-243d-4769-afc9-19e5e4097ef9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:50.928327 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928172 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6f9aa963-3cff-41d4-925c-79ba5db70d57-cni-binary-copy\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.928327 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928158 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:50.928327 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928210 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66fd7\" (UniqueName: \"kubernetes.io/projected/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-kube-api-access-66fd7\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:50.928327 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928259 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-iptables-alerter-script\") pod \"iptables-alerter-265mk\" (UID: \"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f\") " pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:50.928830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928410 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-host\") pod \"node-ca-pbsvn\" (UID: \"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be\") " pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:50.928830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928449 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-systemd\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.928830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928484 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-ovn\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.928830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928525 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-cni-netd\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.928830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928558 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovn-node-metrics-cert\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.928830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928572 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-iptables-alerter-script\") pod \"iptables-alerter-265mk\" (UID: \"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f\") " pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:50.928830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928584 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21d827a7-6f18-4be7-86eb-b9141d12a44f-data-volume\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:50.928830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928656 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/21d827a7-6f18-4be7-86eb-b9141d12a44f-data-volume\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:50.928830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928679 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wgr44\" (UID: \"a1e1a5f5-7d62-43a9-9a51-1cb702e80963\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:50.928830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928699 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-registration-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:50.928830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928715 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:50.928830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928753 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c282411-f8cb-4c83-8959-52035df7cd24-serving-cert\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:50.928830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928791 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovn-node-metrics-cert\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.928830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928776 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-system-cni-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.928830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928828 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-sys\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928868 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wgr44\" (UID: \"a1e1a5f5-7d62-43a9-9a51-1cb702e80963\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928866 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-trusted-ca\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928903 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c282411-f8cb-4c83-8959-52035df7cd24-serving-cert\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928913 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-serving-cert\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928936 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-host-slash\") pod \"iptables-alerter-265mk\" (UID: \"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f\") " pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928953 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkcw\" (UniqueName: \"kubernetes.io/projected/ed9c43e8-604b-44f5-a20c-a3af86848c2c-kube-api-access-ngkcw\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928967 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-ca-trust-extracted\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928993 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.928997 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-serving-cert\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929022 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnbxr\" (UniqueName: \"kubernetes.io/projected/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-api-access-lnbxr\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929034 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-ca-trust-extracted\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929123 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-installation-pull-secrets\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929204 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-cnibin\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929225 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtkhz\" (UniqueName: \"kubernetes.io/projected/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-kube-api-access-vtkhz\") pod \"iptables-alerter-265mk\" (UID: \"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f\") " pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929208 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-trusted-ca\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929263 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsjwr\" (UniqueName: \"kubernetes.io/projected/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-kube-api-access-lsjwr\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929282 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f79b5c8d-c736-4db0-93b4-b6a40ad58955-config-volume\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929297 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxmwp\" (UniqueName: \"kubernetes.io/projected/f79b5c8d-c736-4db0-93b4-b6a40ad58955-kube-api-access-gxmwp\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929318 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-installation-pull-secrets\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929418 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929437 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f79b5c8d-c736-4db0-93b4-b6a40ad58955-config-volume\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:50.929471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929469 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt2sw\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-kube-api-access-zt2sw\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929496 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-run\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929532 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcgvd\" (UniqueName: \"kubernetes.io/projected/4dfa9a2f-f22b-4c85-a594-903928b2034a-kube-api-access-dcgvd\") pod \"ingress-canary-mfwcr\" (UID: \"4dfa9a2f-f22b-4c85-a594-903928b2034a\") " pod="openshift-ingress-canary/ingress-canary-mfwcr" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929558 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc6nb\" (UniqueName: \"kubernetes.io/projected/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-kube-api-access-fc6nb\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929585 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-cni-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929618 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929640 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-socket-dir-parent\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929710 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovnkube-script-lib\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929759 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wgr44\" (UID: \"a1e1a5f5-7d62-43a9-9a51-1cb702e80963\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929779 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929795 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-certificates\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929811 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4922dc9-bd15-45a0-a9f6-e61089e6096b-config\") pod \"service-ca-operator-544d749c8c-p79fm\" (UID: \"a4922dc9-bd15-45a0-a9f6-e61089e6096b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929837 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt58k\" (UniqueName: \"kubernetes.io/projected/cff25808-bc4e-4bd3-b38c-5503a00d45fa-kube-api-access-rt58k\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929863 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-kubernetes\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929884 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-etc-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929885 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovnkube-script-lib\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929928 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tttc4\" (UniqueName: \"kubernetes.io/projected/58c9bbd6-243d-4769-afc9-19e5e4097ef9-kube-api-access-tttc4\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929948 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-certificates\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929961 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-image-registry-private-configuration\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.929988 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-daemon-config\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930005 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4922dc9-bd15-45a0-a9f6-e61089e6096b-config\") pod \"service-ca-operator-544d749c8c-p79fm\" (UID: \"a4922dc9-bd15-45a0-a9f6-e61089e6096b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930015 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6089204f-1338-4875-b311-2fa1b27a03d5-konnectivity-ca\") pod \"konnectivity-agent-gjhv5\" (UID: \"6089204f-1338-4875-b311-2fa1b27a03d5\") " pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930012 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wgr44\" (UID: \"a1e1a5f5-7d62-43a9-9a51-1cb702e80963\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930031 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930052 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh7x7\" (UniqueName: \"kubernetes.io/projected/c8ddcddb-5053-4950-bc4e-5d8f83ec8f08-kube-api-access-qh7x7\") pod \"kas-connection-checker-685895c7c9-9b9bn\" (UID: \"c8ddcddb-5053-4950-bc4e-5d8f83ec8f08\") " pod="kube-system/kas-connection-checker-685895c7c9-9b9bn" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930099 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q6vt\" (UniqueName: \"kubernetes.io/projected/12874a88-98a6-436e-973d-cb931fc769de-kube-api-access-8q6vt\") pod \"network-check-source-6f78d59f99-sj6h8\" (UID: \"12874a88-98a6-436e-973d-cb931fc769de\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sj6h8" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930128 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovnkube-config\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930155 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930179 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-kubelet\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930198 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-image-registry-private-configuration\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930203 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-trusted-ca\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930232 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-daemon-config\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930264 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6089204f-1338-4875-b311-2fa1b27a03d5-konnectivity-ca\") pod \"konnectivity-agent-gjhv5\" (UID: \"6089204f-1338-4875-b311-2fa1b27a03d5\") " pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930307 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d0df1a2-2140-46d5-8662-1148296fb5a8-hosts-file\") pod \"node-resolver-6xn9h\" (UID: \"1d0df1a2-2140-46d5-8662-1148296fb5a8\") " pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930346 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ed9c43e8-604b-44f5-a20c-a3af86848c2c-ovnkube-config\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930348 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf7x5\" (UniqueName: \"kubernetes.io/projected/49aa267f-cf95-4d9b-98c4-df106fcd9620-kube-api-access-nf7x5\") pod \"kas-connection-checker-685895c7c9-2sc7w\" (UID: \"49aa267f-cf95-4d9b-98c4-df106fcd9620\") " pod="kube-system/kas-connection-checker-685895c7c9-2sc7w" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930379 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930414 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb9bd726-ba59-4c55-838f-e00276ca9bef-kubelet-config\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930443 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/78ae398a-fdb1-48f8-94fa-51eb582e663a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6xx9g\" (UID: \"78ae398a-fdb1-48f8-94fa-51eb582e663a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:50.930485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930475 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930532 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-trusted-ca\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930590 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dfa9a2f-f22b-4c85-a594-903928b2034a-cert\") pod \"ingress-canary-mfwcr\" (UID: \"4dfa9a2f-f22b-4c85-a594-903928b2034a\") " pod="openshift-ingress-canary/ingress-canary-mfwcr" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930630 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-systemd-units\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930648 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-cni-bin\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930649 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/78ae398a-fdb1-48f8-94fa-51eb582e663a-nginx-conf\") pod \"networking-console-plugin-86c478bb88-6xx9g\" (UID: \"78ae398a-fdb1-48f8-94fa-51eb582e663a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930665 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-image-registry-private-configuration\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930740 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cnibin\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930808 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4dfa9a2f-f22b-4c85-a594-903928b2034a-cert\") pod \"ingress-canary-mfwcr\" (UID: \"4dfa9a2f-f22b-4c85-a594-903928b2034a\") " pod="openshift-ingress-canary/ingress-canary-mfwcr" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930824 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-stats-auth\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930842 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysctl-d\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930861 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t929j\" (UniqueName: \"kubernetes.io/projected/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-kube-api-access-t929j\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930879 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-certificates\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930908 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-image-registry-private-configuration\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930894 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-bound-sa-token\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930963 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-cni-bin\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.930998 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-stats-auth\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.931042 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-hostroot\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.931117 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9pp4\" (UniqueName: \"kubernetes.io/projected/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-kube-api-access-f9pp4\") pod \"node-ca-pbsvn\" (UID: \"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be\") " pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.931129 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-certificates\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.931155 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.931185 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-device-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.931220 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-trusted-ca\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.931246 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-conf-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.931286 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmvc\" (UniqueName: \"kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc\") pod \"network-check-target-hptcz\" (UID: \"ddae22b7-ddc1-4766-9af6-3899f1edf48c\") " pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:50.931977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.931508 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-trusted-ca\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:50.945393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.945341 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2zxc\" (UniqueName: \"kubernetes.io/projected/a1e1a5f5-7d62-43a9-9a51-1cb702e80963-kube-api-access-n2zxc\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-wgr44\" (UID: \"a1e1a5f5-7d62-43a9-9a51-1cb702e80963\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" Oct 30 16:22:50.945510 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.945490 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-bound-sa-token\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:50.945663 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.945650 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kghb4\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-kube-api-access-kghb4\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:50.945740 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.945702 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzxpj\" (UniqueName: \"kubernetes.io/projected/1d0df1a2-2140-46d5-8662-1148296fb5a8-kube-api-access-gzxpj\") pod \"node-resolver-6xn9h\" (UID: \"1d0df1a2-2140-46d5-8662-1148296fb5a8\") " pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:50.945785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.945702 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqhlw\" (UniqueName: \"kubernetes.io/projected/a21d3561-30d9-4a48-816b-5b7be4e5813d-kube-api-access-lqhlw\") pod \"volume-data-source-validator-58c9f5f46d-rmgmf\" (UID: \"a21d3561-30d9-4a48-816b-5b7be4e5813d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rmgmf" Oct 30 16:22:50.952187 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.952167 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64mgb\" (UniqueName: \"kubernetes.io/projected/a7e4f934-2451-4133-aa1a-ee1145ac2c41-kube-api-access-64mgb\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:50.953675 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.953661 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtkhz\" (UniqueName: \"kubernetes.io/projected/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-kube-api-access-vtkhz\") pod \"iptables-alerter-265mk\" (UID: \"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f\") " pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:50.955035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.955019 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgr65\" (UniqueName: \"kubernetes.io/projected/9c282411-f8cb-4c83-8959-52035df7cd24-kube-api-access-cgr65\") pod \"insights-operator-7c4986d449-kx55q\" (UID: \"9c282411-f8cb-4c83-8959-52035df7cd24\") " pod="openshift-insights/insights-operator-7c4986d449-kx55q" Oct 30 16:22:50.955080 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.955047 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qglw\" (UniqueName: \"kubernetes.io/projected/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-kube-api-access-4qglw\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:50.956462 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.956441 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt58k\" (UniqueName: \"kubernetes.io/projected/cff25808-bc4e-4bd3-b38c-5503a00d45fa-kube-api-access-rt58k\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:50.956528 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.956508 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxmwp\" (UniqueName: \"kubernetes.io/projected/f79b5c8d-c736-4db0-93b4-b6a40ad58955-kube-api-access-gxmwp\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:50.956566 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.956551 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc6nb\" (UniqueName: \"kubernetes.io/projected/5a25e3b8-4f22-4fdb-86a9-70d1c594236d-kube-api-access-fc6nb\") pod \"console-operator-7cd56b8dbc-szx5s\" (UID: \"5a25e3b8-4f22-4fdb-86a9-70d1c594236d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:50.956605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.956561 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwpb8\" (UniqueName: \"kubernetes.io/projected/1026393f-ba8e-408a-b7df-a1d03e047b1f-kube-api-access-nwpb8\") pod \"kas-connection-checker-685895c7c9-5r74t\" (UID: \"1026393f-ba8e-408a-b7df-a1d03e047b1f\") " pod="kube-system/kas-connection-checker-685895c7c9-5r74t" Oct 30 16:22:50.956646 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.956604 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jmvc\" (UniqueName: \"kubernetes.io/projected/ddae22b7-ddc1-4766-9af6-3899f1edf48c-kube-api-access-2jmvc\") pod \"network-check-target-hptcz\" (UID: \"ddae22b7-ddc1-4766-9af6-3899f1edf48c\") " pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:50.958991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.958975 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnbxr\" (UniqueName: \"kubernetes.io/projected/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-api-access-lnbxr\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:50.959884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.959866 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldgcm\" (UniqueName: \"kubernetes.io/projected/a4922dc9-bd15-45a0-a9f6-e61089e6096b-kube-api-access-ldgcm\") pod \"service-ca-operator-544d749c8c-p79fm\" (UID: \"a4922dc9-bd15-45a0-a9f6-e61089e6096b\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" Oct 30 16:22:50.960174 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.960156 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcgvd\" (UniqueName: \"kubernetes.io/projected/4dfa9a2f-f22b-4c85-a594-903928b2034a-kube-api-access-dcgvd\") pod \"ingress-canary-mfwcr\" (UID: \"4dfa9a2f-f22b-4c85-a594-903928b2034a\") " pod="openshift-ingress-canary/ingress-canary-mfwcr" Oct 30 16:22:50.961049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.961031 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9pp4\" (UniqueName: \"kubernetes.io/projected/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-kube-api-access-f9pp4\") pod \"node-ca-pbsvn\" (UID: \"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be\") " pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:50.961128 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.961111 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngkcw\" (UniqueName: \"kubernetes.io/projected/ed9c43e8-604b-44f5-a20c-a3af86848c2c-kube-api-access-ngkcw\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:50.961208 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.961189 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tttc4\" (UniqueName: \"kubernetes.io/projected/58c9bbd6-243d-4769-afc9-19e5e4097ef9-kube-api-access-tttc4\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:50.961347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.961328 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xlkr\" (UniqueName: \"kubernetes.io/projected/6f9aa963-3cff-41d4-925c-79ba5db70d57-kube-api-access-9xlkr\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:50.961432 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.961419 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsjwr\" (UniqueName: \"kubernetes.io/projected/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-kube-api-access-lsjwr\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:50.961832 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.961813 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q6vt\" (UniqueName: \"kubernetes.io/projected/12874a88-98a6-436e-973d-cb931fc769de-kube-api-access-8q6vt\") pod \"network-check-source-6f78d59f99-sj6h8\" (UID: \"12874a88-98a6-436e-973d-cb931fc769de\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sj6h8" Oct 30 16:22:50.962991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.962976 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t929j\" (UniqueName: \"kubernetes.io/projected/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-kube-api-access-t929j\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:50.964098 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.964081 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf7x5\" (UniqueName: \"kubernetes.io/projected/49aa267f-cf95-4d9b-98c4-df106fcd9620-kube-api-access-nf7x5\") pod \"kas-connection-checker-685895c7c9-2sc7w\" (UID: \"49aa267f-cf95-4d9b-98c4-df106fcd9620\") " pod="kube-system/kas-connection-checker-685895c7c9-2sc7w" Oct 30 16:22:50.964448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.964433 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt2sw\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-kube-api-access-zt2sw\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:50.965140 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.965126 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-bound-sa-token\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:50.965506 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.965491 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66fd7\" (UniqueName: \"kubernetes.io/projected/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-kube-api-access-66fd7\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:50.966289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:50.966271 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh7x7\" (UniqueName: \"kubernetes.io/projected/c8ddcddb-5053-4950-bc4e-5d8f83ec8f08-kube-api-access-qh7x7\") pod \"kas-connection-checker-685895c7c9-9b9bn\" (UID: \"c8ddcddb-5053-4950-bc4e-5d8f83ec8f08\") " pod="kube-system/kas-connection-checker-685895c7c9-9b9bn" Oct 30 16:22:51.011151 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.011131 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:51.031996 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.031972 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:51.032082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032002 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-kubelet\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.032082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032025 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d0df1a2-2140-46d5-8662-1148296fb5a8-hosts-file\") pod \"node-resolver-6xn9h\" (UID: \"1d0df1a2-2140-46d5-8662-1148296fb5a8\") " pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:51.032082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032049 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:51.032082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032072 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb9bd726-ba59-4c55-838f-e00276ca9bef-kubelet-config\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:51.032263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032084 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-kubelet\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.032263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032096 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:51.032263 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.032115 4861 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:22:51.032263 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.032133 4861 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:22:51.032263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032136 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cb9bd726-ba59-4c55-838f-e00276ca9bef-kubelet-config\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:51.032263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032115 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d0df1a2-2140-46d5-8662-1148296fb5a8-hosts-file\") pod \"node-resolver-6xn9h\" (UID: \"1d0df1a2-2140-46d5-8662-1148296fb5a8\") " pod="openshift-dns/node-resolver-6xn9h" Oct 30 16:22:51.032263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032156 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-systemd-units\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.032263 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.032171 4861 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:22:51.032263 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.032135 4861 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dfdc4b7f-lkk6g: secret "image-registry-tls" not found Oct 30 16:22:51.032263 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.032200 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs podName:1ab95e94-bdff-4f1f-a0fe-095a2509d6f8 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:51.532185624 +0000 UTC m=+1.874859657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs") pod "router-default-58c95ddf47-6m2jw" (UID: "1ab95e94-bdff-4f1f-a0fe-095a2509d6f8") : secret "router-metrics-certs-default" not found Oct 30 16:22:51.032263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032121 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-systemd-units\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.032263 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.032257 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls podName:21d827a7-6f18-4be7-86eb-b9141d12a44f nodeName:}" failed. No retries permitted until 2026-10-30 16:22:51.532217732 +0000 UTC m=+1.874891756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnmqt" (UID: "21d827a7-6f18-4be7-86eb-b9141d12a44f") : secret "insights-runtime-extractor-tls" not found Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.032299 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls podName:7932d7cc-b9fe-423f-bc88-93711eaec908 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:51.532282416 +0000 UTC m=+1.874956441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls") pod "image-registry-5dfdc4b7f-lkk6g" (UID: "7932d7cc-b9fe-423f-bc88-93711eaec908") : secret "image-registry-tls" not found Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032319 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-cni-bin\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032360 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-cni-bin\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032365 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cnibin\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032389 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-cnibin\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032393 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysctl-d\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032426 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-cni-bin\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032448 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-hostroot\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032470 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032476 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysctl-d\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032492 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-device-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032500 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-cni-bin\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032512 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-hostroot\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032527 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-conf-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032550 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032568 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032587 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysconfig\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032549 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-device-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032604 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-lib-modules\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032607 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032610 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-conf-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.032607 4861 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032627 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-node-log\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.032652 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls podName:f79b5c8d-c736-4db0-93b4-b6a40ad58955 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:51.532643538 +0000 UTC m=+1.875317554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls") pod "dns-default-82w6j" (UID: "f79b5c8d-c736-4db0-93b4-b6a40ad58955") : secret "dns-default-metrics-tls" not found Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032665 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-system-cni-dir\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032677 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-node-log\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032665 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysconfig\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032691 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-etc-kubernetes\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032704 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-system-cni-dir\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032712 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-lib-modules\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032737 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-etc-kubernetes\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032739 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-modprobe-d\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032790 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-etc-selinux\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.032801 4861 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032820 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-socket-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.032839 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls podName:7fff2d07-9f9e-4edb-a32c-ad3a9440fc17 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:51.532829001 +0000 UTC m=+1.875503017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-btqzf" (UID: "7fff2d07-9f9e-4edb-a32c-ad3a9440fc17") : secret "samples-operator-tls" not found Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032859 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-cni-multus\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.032841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032867 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-socket-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032804 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-modprobe-d\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032877 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032907 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-etc-selinux\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032914 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032914 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-var-lib-kubelet\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032918 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-var-lib-cni-multus\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032947 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-var-lib-kubelet\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.032977 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033016 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-sys-fs\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033046 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033057 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-sys-fs\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033010 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-run-ovn-kubernetes\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033078 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033109 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21d827a7-6f18-4be7-86eb-b9141d12a44f-crio-socket\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.033127 4861 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033137 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.033142 4861 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.033155 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls podName:93b8a9b8-43df-4dc0-97da-4bfd11f373cd nodeName:}" failed. No retries permitted until 2026-10-30 16:22:51.533146516 +0000 UTC m=+1.875820533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gn2hp" (UID: "93b8a9b8-43df-4dc0-97da-4bfd11f373cd") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033178 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb9bd726-ba59-4c55-838f-e00276ca9bef-dbus\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.033184 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs podName:cff25808-bc4e-4bd3-b38c-5503a00d45fa nodeName:}" failed. No retries permitted until 2026-10-30 16:22:51.533171176 +0000 UTC m=+1.875845192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs") pod "network-metrics-daemon-fz7kn" (UID: "cff25808-bc4e-4bd3-b38c-5503a00d45fa") : secret "metrics-daemon-secret" not found Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033200 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/21d827a7-6f18-4be7-86eb-b9141d12a44f-crio-socket\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033219 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6xx9g\" (UID: \"78ae398a-fdb1-48f8-94fa-51eb582e663a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.033227 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle podName:1ab95e94-bdff-4f1f-a0fe-095a2509d6f8 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:51.533217303 +0000 UTC m=+1.875891354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle") pod "router-default-58c95ddf47-6m2jw" (UID: "1ab95e94-bdff-4f1f-a0fe-095a2509d6f8") : configmap references non-existent config key: service-ca.crt Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033228 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cb9bd726-ba59-4c55-838f-e00276ca9bef-dbus\") pod \"global-pull-secret-syncer-pq64k\" (UID: \"cb9bd726-ba59-4c55-838f-e00276ca9bef\") " pod="kube-system/global-pull-secret-syncer-pq64k" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033254 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-host\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.033257 4861 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033271 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-netns\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.033288 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert podName:78ae398a-fdb1-48f8-94fa-51eb582e663a nodeName:}" failed. No retries permitted until 2026-10-30 16:22:51.5332748 +0000 UTC m=+1.875948817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6xx9g" (UID: "78ae398a-fdb1-48f8-94fa-51eb582e663a") : secret "networking-console-plugin-cert" not found Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033304 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-netns\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033307 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-var-lib-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033327 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033327 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-var-lib-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033359 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-host\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033383 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58c9bbd6-243d-4769-afc9-19e5e4097ef9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7stpt\" (UID: \"58c9bbd6-243d-4769-afc9-19e5e4097ef9\") " pod="openshift-multus/multus-additional-cni-plugins-7stpt" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033382 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-multus-certs\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033420 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-systemd\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033436 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysctl-conf\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033452 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-run-netns\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033473 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-log-socket\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033484 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-systemd\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033488 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-kubelet\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033514 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-run-netns\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033518 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-multus-certs\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033521 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-k8s-cni-cncf-io\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033534 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-log-socket\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033537 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-kubelet\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033550 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-host-run-k8s-cni-cncf-io\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033556 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-slash\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033584 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-sysctl-conf\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033584 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-slash\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033583 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033637 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-host\") pod \"node-ca-pbsvn\" (UID: \"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be\") " pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033663 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-systemd\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033676 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-ovn\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033689 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-cni-netd\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033714 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-registration-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033761 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-ovn\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033763 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033773 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9ab0447a-bdef-45aa-92c3-e9d35ee9c2be-host\") pod \"node-ca-pbsvn\" (UID: \"9ab0447a-bdef-45aa-92c3-e9d35ee9c2be\") " pod="openshift-image-registry/node-ca-pbsvn" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033771 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-run-systemd\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033791 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-system-cni-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033795 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-host-cni-netd\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033812 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-sys\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033820 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-registration-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033838 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-host-slash\") pod \"iptables-alerter-265mk\" (UID: \"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f\") " pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033844 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-system-cni-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.033849 4861 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033865 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033871 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f-host-slash\") pod \"iptables-alerter-265mk\" (UID: \"a890f0ad-4cdf-4bb3-b2b5-7ba2caf6802f\") " pod="openshift-network-operator/iptables-alerter-265mk" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.033871 4861 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8: secret "image-registry-tls" not found Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033889 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-sys\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033892 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-cnibin\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033905 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a7e4f934-2451-4133-aa1a-ee1145ac2c41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dctc7\" (UID: \"a7e4f934-2451-4133-aa1a-ee1145ac2c41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dctc7" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033929 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-cnibin\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.033936 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls podName:fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:51.533921989 +0000 UTC m=+1.876596008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls") pod "image-registry-5cc8d7fd6b-v2gt8" (UID: "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5") : secret "image-registry-tls" not found Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033953 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-run\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033980 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-run\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.033982 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-cni-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.034013 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-socket-dir-parent\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.034029 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-cni-dir\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.034063 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-kubernetes\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.034080 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-etc-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.034043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.034086 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6f9aa963-3cff-41d4-925c-79ba5db70d57-multus-socket-dir-parent\") pod \"multus-wr6tc\" (UID: \"6f9aa963-3cff-41d4-925c-79ba5db70d57\") " pod="openshift-multus/multus-wr6tc" Oct 30 16:22:51.036781 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.034075 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/21d827a7-6f18-4be7-86eb-b9141d12a44f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:51.036781 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.034122 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f4d6719-619b-46dd-9b1d-93fd5d1e91c0-etc-kubernetes\") pod \"tuned-rwx47\" (UID: \"2f4d6719-619b-46dd-9b1d-93fd5d1e91c0\") " pod="openshift-cluster-node-tuning-operator/tuned-rwx47" Oct 30 16:22:51.036781 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.034143 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ed9c43e8-604b-44f5-a20c-a3af86848c2c-etc-openvswitch\") pod \"ovnkube-node-l95sm\" (UID: \"ed9c43e8-604b-44f5-a20c-a3af86848c2c\") " pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:51.098045 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.098020 4861 scope.go:122] "RemoveContainer" containerID="9327de775d1ac0e11a7137ae4ef9fb291b5a69127a8688d920be8a7a4ccfe5fa" Oct 30 16:22:51.432416 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.432386 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:51.539030 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.538957 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:51.539030 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.538989 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:51.539030 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.539010 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:51.539030 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.539028 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6xx9g\" (UID: \"78ae398a-fdb1-48f8-94fa-51eb582e663a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:51.539322 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.539060 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:51.539322 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.539105 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:51.539322 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.539123 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:51.539322 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.539123 4861 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:51.539322 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.539131 4861 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 16:22:51.539322 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.539142 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:51.539322 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.539153 4861 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 16:22:51.539322 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.539176 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:51.539322 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.539207 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls podName:93b8a9b8-43df-4dc0-97da-4bfd11f373cd nodeName:}" failed. No retries permitted until 2026-10-30 16:22:52.539182046 +0000 UTC m=+2.881856081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gn2hp" (UID: "93b8a9b8-43df-4dc0-97da-4bfd11f373cd") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:51.539767 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.539358 4861 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:22:51.539767 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.539414 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert podName:78ae398a-fdb1-48f8-94fa-51eb582e663a nodeName:}" failed. No retries permitted until 2026-10-30 16:22:52.539396966 +0000 UTC m=+2.882070981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-6xx9g" (UID: "78ae398a-fdb1-48f8-94fa-51eb582e663a") : secret "networking-console-plugin-cert" not found Oct 30 16:22:51.539767 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.539437 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs podName:cff25808-bc4e-4bd3-b38c-5503a00d45fa nodeName:}" failed. No retries permitted until 2026-10-30 16:22:52.539426128 +0000 UTC m=+2.882100167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs") pod "network-metrics-daemon-fz7kn" (UID: "cff25808-bc4e-4bd3-b38c-5503a00d45fa") : secret "metrics-daemon-secret" not found Oct 30 16:22:51.539767 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.539447 4861 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:22:51.539767 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.539455 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs podName:1ab95e94-bdff-4f1f-a0fe-095a2509d6f8 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:52.539446545 +0000 UTC m=+2.882120561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs") pod "router-default-58c95ddf47-6m2jw" (UID: "1ab95e94-bdff-4f1f-a0fe-095a2509d6f8") : secret "router-metrics-certs-default" not found Oct 30 16:22:51.539767 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.539488 4861 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 16:22:51.539767 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.539494 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:51.539767 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.539503 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls podName:21d827a7-6f18-4be7-86eb-b9141d12a44f nodeName:}" failed. No retries permitted until 2026-10-30 16:22:52.539486763 +0000 UTC m=+2.882160793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnmqt" (UID: "21d827a7-6f18-4be7-86eb-b9141d12a44f") : secret "insights-runtime-extractor-tls" not found Oct 30 16:22:51.539767 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.539558 4861 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 16:22:51.539767 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.539596 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls podName:7fff2d07-9f9e-4edb-a32c-ad3a9440fc17 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:52.539585103 +0000 UTC m=+2.882259121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-btqzf" (UID: "7fff2d07-9f9e-4edb-a32c-ad3a9440fc17") : secret "samples-operator-tls" not found Oct 30 16:22:51.540455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.539849 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-service-ca-bundle\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:51.541708 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.541684 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f79b5c8d-c736-4db0-93b4-b6a40ad58955-metrics-tls\") pod \"dns-default-82w6j\" (UID: \"f79b5c8d-c736-4db0-93b4-b6a40ad58955\") " pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:51.541819 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.541793 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls\") pod \"image-registry-5cc8d7fd6b-v2gt8\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:51.541855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.541845 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls\") pod \"image-registry-5dfdc4b7f-lkk6g\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:51.696686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.696664 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:22:51.697164 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.697140 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:51.704320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.704303 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:51.707441 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.707425 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:51.857405 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.857380 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:22:51.857936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.857919 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/1.log" Oct 30 16:22:51.858032 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.857957 4861 generic.go:356] "Generic (PLEG): container finished" podID="5a25e3b8-4f22-4fdb-86a9-70d1c594236d" containerID="2d34b10a6545d4f5ec331e68b524f96cae91d7720553ea8ffb753f4e05030d9c" exitCode=255 Oct 30 16:22:51.858083 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.858027 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" event={"ID":"5a25e3b8-4f22-4fdb-86a9-70d1c594236d","Type":"ContainerDied","Data":"2d34b10a6545d4f5ec331e68b524f96cae91d7720553ea8ffb753f4e05030d9c"} Oct 30 16:22:51.858083 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.858078 4861 scope.go:122] "RemoveContainer" containerID="9327de775d1ac0e11a7137ae4ef9fb291b5a69127a8688d920be8a7a4ccfe5fa" Oct 30 16:22:51.858669 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.858645 4861 scope.go:122] "RemoveContainer" containerID="2d34b10a6545d4f5ec331e68b524f96cae91d7720553ea8ffb753f4e05030d9c" Oct 30 16:22:51.858874 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:51.858854 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-szx5s_openshift-console-operator(5a25e3b8-4f22-4fdb-86a9-70d1c594236d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" podUID="5a25e3b8-4f22-4fdb-86a9-70d1c594236d" Oct 30 16:22:51.894146 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:51.894092 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:52.035584 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.035553 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dfdc4b7f-lkk6g"] Oct 30 16:22:52.040948 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:52.040920 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7932d7cc_b9fe_423f_bc88_93711eaec908.slice/crio-89aa7b4b39305f8d331379e508c24305d4c35ab972e67721f438e3a00b540292 WatchSource:0}: Error finding container 89aa7b4b39305f8d331379e508c24305d4c35ab972e67721f438e3a00b540292: Status 404 returned error can't find the container with id 89aa7b4b39305f8d331379e508c24305d4c35ab972e67721f438e3a00b540292 Oct 30 16:22:52.054685 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.054663 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8"] Oct 30 16:22:52.058686 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:52.058664 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7c4e1f_a6c4_4303_8e1b_d9c043c62fc5.slice/crio-ac5236464d8d1aef2df6c4a33522ca8940758c7cdb7918af7cc5a69048d9de1c WatchSource:0}: Error finding container ac5236464d8d1aef2df6c4a33522ca8940758c7cdb7918af7cc5a69048d9de1c: Status 404 returned error can't find the container with id ac5236464d8d1aef2df6c4a33522ca8940758c7cdb7918af7cc5a69048d9de1c Oct 30 16:22:52.084698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.084627 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-82w6j"] Oct 30 16:22:52.087734 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:52.087698 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79b5c8d_c736_4db0_93b4_b6a40ad58955.slice/crio-0fd4b767da3acec15b4be7f7a868b25de214a83da42d8f468917857ca755bf77 WatchSource:0}: Error finding container 0fd4b767da3acec15b4be7f7a868b25de214a83da42d8f468917857ca755bf77: Status 404 returned error can't find the container with id 0fd4b767da3acec15b4be7f7a868b25de214a83da42d8f468917857ca755bf77 Oct 30 16:22:52.089938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.089899 4861 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:22:52.223718 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.223690 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:52.366855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.366773 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:52.402902 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.402874 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:52.548979 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.548942 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:52.549139 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.549010 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:52.549139 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.549041 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:52.549139 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.549076 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6xx9g\" (UID: \"78ae398a-fdb1-48f8-94fa-51eb582e663a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:52.549311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.549193 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:52.549311 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:52.549193 4861 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:52.549311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.549211 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:52.549311 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:52.549267 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls podName:93b8a9b8-43df-4dc0-97da-4bfd11f373cd nodeName:}" failed. No retries permitted until 2026-10-30 16:22:54.549251516 +0000 UTC m=+4.891925547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gn2hp" (UID: "93b8a9b8-43df-4dc0-97da-4bfd11f373cd") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:52.549524 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:52.549319 4861 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:22:52.549524 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:52.549373 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls podName:21d827a7-6f18-4be7-86eb-b9141d12a44f nodeName:}" failed. No retries permitted until 2026-10-30 16:22:54.549356838 +0000 UTC m=+4.892030868 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnmqt" (UID: "21d827a7-6f18-4be7-86eb-b9141d12a44f") : secret "insights-runtime-extractor-tls" not found Oct 30 16:22:52.550705 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:52.550672 4861 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:22:52.550844 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:52.550779 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs podName:1ab95e94-bdff-4f1f-a0fe-095a2509d6f8 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:54.550750672 +0000 UTC m=+4.893424705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs") pod "router-default-58c95ddf47-6m2jw" (UID: "1ab95e94-bdff-4f1f-a0fe-095a2509d6f8") : secret "router-metrics-certs-default" not found Oct 30 16:22:52.552117 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.552089 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cff25808-bc4e-4bd3-b38c-5503a00d45fa-metrics-certs\") pod \"network-metrics-daemon-fz7kn\" (UID: \"cff25808-bc4e-4bd3-b38c-5503a00d45fa\") " pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:52.552229 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.552213 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7fff2d07-9f9e-4edb-a32c-ad3a9440fc17-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-btqzf\" (UID: \"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:52.556387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.556362 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/78ae398a-fdb1-48f8-94fa-51eb582e663a-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-6xx9g\" (UID: \"78ae398a-fdb1-48f8-94fa-51eb582e663a\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:52.597633 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.597606 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" Oct 30 16:22:52.599767 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.599744 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" Oct 30 16:22:52.604612 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.604593 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz7kn" Oct 30 16:22:52.864743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.864209 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" event={"ID":"7932d7cc-b9fe-423f-bc88-93711eaec908","Type":"ContainerStarted","Data":"b9dc88d5b6e276503f81273536b8ff605b12ff8f2451b40d0d0cb302a6eb4156"} Oct 30 16:22:52.864743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.864305 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" event={"ID":"7932d7cc-b9fe-423f-bc88-93711eaec908","Type":"ContainerStarted","Data":"89aa7b4b39305f8d331379e508c24305d4c35ab972e67721f438e3a00b540292"} Oct 30 16:22:52.866131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.866098 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" event={"ID":"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5","Type":"ContainerStarted","Data":"cea1272d07d14025d4d8bea0b59d2b86792c7751c6281122c326225f54f5d484"} Oct 30 16:22:52.866243 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.866143 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" event={"ID":"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5","Type":"ContainerStarted","Data":"ac5236464d8d1aef2df6c4a33522ca8940758c7cdb7918af7cc5a69048d9de1c"} Oct 30 16:22:52.868072 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.868022 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:22:52.869404 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.869382 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-82w6j" event={"ID":"f79b5c8d-c736-4db0-93b4-b6a40ad58955","Type":"ContainerStarted","Data":"0fd4b767da3acec15b4be7f7a868b25de214a83da42d8f468917857ca755bf77"} Oct 30 16:22:52.928411 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.928335 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf"] Oct 30 16:22:52.936752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.936711 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fz7kn"] Oct 30 16:22:52.939631 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.939610 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-6xx9g"] Oct 30 16:22:52.949123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:52.949100 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ae398a_fdb1_48f8_94fa_51eb582e663a.slice/crio-cdfcd313eecdf005d080d087bb6168b8c2d06e1c818ff6fa0351b18f6176deea WatchSource:0}: Error finding container cdfcd313eecdf005d080d087bb6168b8c2d06e1c818ff6fa0351b18f6176deea: Status 404 returned error can't find the container with id cdfcd313eecdf005d080d087bb6168b8c2d06e1c818ff6fa0351b18f6176deea Oct 30 16:22:52.950455 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:52.950430 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff25808_bc4e_4bd3_b38c_5503a00d45fa.slice/crio-329f80e8de4d979a7b69a4a7774a28d24bad0402048393775effdac729800a46 WatchSource:0}: Error finding container 329f80e8de4d979a7b69a4a7774a28d24bad0402048393775effdac729800a46: Status 404 returned error can't find the container with id 329f80e8de4d979a7b69a4a7774a28d24bad0402048393775effdac729800a46 Oct 30 16:22:52.954212 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:52.954186 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:53.169173 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:53.169112 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:53.325768 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:53.325718 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:53.567883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:53.567812 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:53.732295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:53.732262 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:53.875627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:53.875561 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" event={"ID":"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17","Type":"ContainerStarted","Data":"450143ac8e0cca21827767b760eeba350b80d100695f96fe7e2b7a8ef0ee5b58"} Oct 30 16:22:53.878131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:53.878063 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" event={"ID":"78ae398a-fdb1-48f8-94fa-51eb582e663a","Type":"ContainerStarted","Data":"cdfcd313eecdf005d080d087bb6168b8c2d06e1c818ff6fa0351b18f6176deea"} Oct 30 16:22:53.881017 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:53.880944 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-82w6j" event={"ID":"f79b5c8d-c736-4db0-93b4-b6a40ad58955","Type":"ContainerStarted","Data":"a41421f15bd925a2c109ee5a35b30dc4ff1bcb5495af9bf0c55a70b67987d1d2"} Oct 30 16:22:53.883714 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:53.882666 4861 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:22:53.883714 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:53.883221 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fz7kn" event={"ID":"cff25808-bc4e-4bd3-b38c-5503a00d45fa","Type":"ContainerStarted","Data":"329f80e8de4d979a7b69a4a7774a28d24bad0402048393775effdac729800a46"} Oct 30 16:22:53.883714 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:53.883291 4861 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:22:54.344886 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:54.344865 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:54.570353 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:54.570311 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:54.570533 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:54.570364 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:54.570533 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:54.570424 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:54.570533 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:54.570439 4861 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 16:22:54.570533 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:54.570504 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs podName:1ab95e94-bdff-4f1f-a0fe-095a2509d6f8 nodeName:}" failed. No retries permitted until 2026-10-30 16:22:58.570485767 +0000 UTC m=+8.913159808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs") pod "router-default-58c95ddf47-6m2jw" (UID: "1ab95e94-bdff-4f1f-a0fe-095a2509d6f8") : secret "router-metrics-certs-default" not found Oct 30 16:22:54.570702 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:54.570545 4861 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:54.570702 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:54.570552 4861 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:22:54.570702 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:54.570581 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls podName:93b8a9b8-43df-4dc0-97da-4bfd11f373cd nodeName:}" failed. No retries permitted until 2026-10-30 16:22:58.570570127 +0000 UTC m=+8.913244161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gn2hp" (UID: "93b8a9b8-43df-4dc0-97da-4bfd11f373cd") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:54.570702 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:54.570626 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls podName:21d827a7-6f18-4be7-86eb-b9141d12a44f nodeName:}" failed. No retries permitted until 2026-10-30 16:22:58.570605682 +0000 UTC m=+8.913279712 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnmqt" (UID: "21d827a7-6f18-4be7-86eb-b9141d12a44f") : secret "insights-runtime-extractor-tls" not found Oct 30 16:22:54.888429 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:54.888389 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-6xx9g" event={"ID":"78ae398a-fdb1-48f8-94fa-51eb582e663a","Type":"ContainerStarted","Data":"95e2f2e0b10f4cfc6951980bbc3cc4d0050b28d014399df2f37cdb3d4d6dd071"} Oct 30 16:22:54.891257 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:54.891230 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-82w6j" event={"ID":"f79b5c8d-c736-4db0-93b4-b6a40ad58955","Type":"ContainerStarted","Data":"d0f15a500ca647040055b15505ec89d4e6669a3248b76927566f7ab9bcf9b711"} Oct 30 16:22:54.891405 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:54.891296 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-82w6j" Oct 30 16:22:54.893430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:54.893400 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fz7kn" event={"ID":"cff25808-bc4e-4bd3-b38c-5503a00d45fa","Type":"ContainerStarted","Data":"293e17aeafe13e9075ea73137011ffe4d50c7538765aee9af6988c26a643c544"} Oct 30 16:22:54.893532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:54.893454 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fz7kn" event={"ID":"cff25808-bc4e-4bd3-b38c-5503a00d45fa","Type":"ContainerStarted","Data":"8f36abe61d9b31943e6cc039853dfb96376b6ce5a0a9148b0244ba3c5ef5638b"} Oct 30 16:22:54.977292 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:54.977259 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:55.131485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:55.131440 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:55.207834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:55.207771 4861 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:22:55.208125 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:55.208112 4861 scope.go:122] "RemoveContainer" containerID="2d34b10a6545d4f5ec331e68b524f96cae91d7720553ea8ffb753f4e05030d9c" Oct 30 16:22:55.208301 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:55.208285 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-szx5s_openshift-console-operator(5a25e3b8-4f22-4fdb-86a9-70d1c594236d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" podUID="5a25e3b8-4f22-4fdb-86a9-70d1c594236d" Oct 30 16:22:55.263939 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:55.263920 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6xn9h_1d0df1a2-2140-46d5-8662-1148296fb5a8/dns-node-resolver/0.log" Oct 30 16:22:55.538005 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:55.537973 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:55.664243 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:55.664216 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5cc8d7fd6b-v2gt8_fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5/registry/0.log" Oct 30 16:22:55.736935 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:55.736903 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:55.863698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:55.863637 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5dfdc4b7f-lkk6g_7932d7cc-b9fe-423f-bc88-93711eaec908/registry/0.log" Oct 30 16:22:55.898523 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:55.898481 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" event={"ID":"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17","Type":"ContainerStarted","Data":"52ea0c2444c36593601de5c89275b5a86ffae7832cdf393b70a1a16c3839556c"} Oct 30 16:22:55.898938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:55.898551 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-btqzf" event={"ID":"7fff2d07-9f9e-4edb-a32c-ad3a9440fc17","Type":"ContainerStarted","Data":"346e1e6dec95f76b2c882e9329fb8feee516b17f4229dedaabbd0936c59c37c1"} Oct 30 16:22:56.134844 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:56.134755 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:56.263559 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:56.263528 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pbsvn_9ab0447a-bdef-45aa-92c3-e9d35ee9c2be/node-ca/0.log" Oct 30 16:22:56.942290 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:56.942255 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:57.264520 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:57.264495 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mfwcr_4dfa9a2f-f22b-4c85-a594-903928b2034a/serve-healthcheck-canary/0.log" Oct 30 16:22:57.345459 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:57.345426 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:57.538323 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:57.538251 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:57.735386 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:57.735354 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:57.865778 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:57.865701 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-wgr44_a1e1a5f5-7d62-43a9-9a51-1cb702e80963/kube-storage-version-migrator-operator/0.log" Oct 30 16:22:57.881324 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:57.881298 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:58.201087 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:58.201007 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:58.305612 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:58.305587 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:22:58.305885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:58.305861 4861 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:22:58.603181 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:58.603142 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:58.603348 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:58.603188 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:22:58.603348 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:58.603237 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:22:58.603478 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:58.603345 4861 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 16:22:58.603478 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:58.603349 4861 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:58.603478 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:58.603406 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls podName:21d827a7-6f18-4be7-86eb-b9141d12a44f nodeName:}" failed. No retries permitted until 2026-10-30 16:23:06.603391945 +0000 UTC m=+16.946065961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnmqt" (UID: "21d827a7-6f18-4be7-86eb-b9141d12a44f") : secret "insights-runtime-extractor-tls" not found Oct 30 16:22:58.603478 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:22:58.603420 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls podName:93b8a9b8-43df-4dc0-97da-4bfd11f373cd nodeName:}" failed. No retries permitted until 2026-10-30 16:23:06.603413515 +0000 UTC m=+16.946087532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-gn2hp" (UID: "93b8a9b8-43df-4dc0-97da-4bfd11f373cd") : secret "cluster-monitoring-operator-tls" not found Oct 30 16:22:58.605669 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:58.605648 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab95e94-bdff-4f1f-a0fe-095a2509d6f8-metrics-certs\") pod \"router-default-58c95ddf47-6m2jw\" (UID: \"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8\") " pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:58.897385 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:58.897296 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:22:58.897385 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:58.897321 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:22:58.937561 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:58.937518 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:58.991030 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:58.990998 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:22:59.143329 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:59.143299 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:59.230103 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:59.230082 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-58c95ddf47-6m2jw"] Oct 30 16:22:59.233341 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:22:59.233315 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab95e94_bdff_4f1f_a0fe_095a2509d6f8.slice/crio-8d5378a557aa05cb13fc6116eaaf27e52867439bd5033be57b195644b830306f WatchSource:0}: Error finding container 8d5378a557aa05cb13fc6116eaaf27e52867439bd5033be57b195644b830306f: Status 404 returned error can't find the container with id 8d5378a557aa05cb13fc6116eaaf27e52867439bd5033be57b195644b830306f Oct 30 16:22:59.330489 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:59.330458 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:59.537476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:59.537405 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:59.681950 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:59.681917 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:59.813747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:59.813661 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:22:59.916793 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:59.916760 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-58c95ddf47-6m2jw" event={"ID":"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8","Type":"ContainerStarted","Data":"9a36b75e5d6d6f87badf6ef3eec506fbe8f8cb306769f8986f74125364a0e80d"} Oct 30 16:22:59.916945 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:59.916824 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-58c95ddf47-6m2jw" event={"ID":"1ab95e94-bdff-4f1f-a0fe-095a2509d6f8","Type":"ContainerStarted","Data":"8d5378a557aa05cb13fc6116eaaf27e52867439bd5033be57b195644b830306f"} Oct 30 16:22:59.954900 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:22:59.954872 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:00.097789 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:00.097707 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:00.241194 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:00.241161 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:00.736910 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:00.736883 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:00.892874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:00.892843 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:23:00.898530 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:00.898509 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:23:00.901015 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:00.900995 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:23:00.990674 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:00.990612 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:23:00.990998 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:00.990981 4861 scope.go:122] "RemoveContainer" containerID="2d34b10a6545d4f5ec331e68b524f96cae91d7720553ea8ffb753f4e05030d9c" Oct 30 16:23:00.991165 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:23:00.991146 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-szx5s_openshift-console-operator(5a25e3b8-4f22-4fdb-86a9-70d1c594236d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" podUID="5a25e3b8-4f22-4fdb-86a9-70d1c594236d" Oct 30 16:23:01.348656 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:01.348584 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:01.535037 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:01.535008 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:01.747801 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:01.747767 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:01.923402 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:01.923369 4861 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:23:01.923773 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:01.923756 4861 scope.go:122] "RemoveContainer" containerID="2d34b10a6545d4f5ec331e68b524f96cae91d7720553ea8ffb753f4e05030d9c" Oct 30 16:23:01.928586 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:01.928563 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:02.084563 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:02.084489 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:02.739059 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:02.739020 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:02.884938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:02.884904 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:02.932175 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:02.932147 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:23:02.932310 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:02.932235 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" event={"ID":"5a25e3b8-4f22-4fdb-86a9-70d1c594236d","Type":"ContainerStarted","Data":"f3635090d8919b5cf4dca23ff3c1398eb265a1c80935e907a41234493bdc009b"} Oct 30 16:23:02.932540 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:02.932520 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:23:02.937133 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:02.937111 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-szx5s" Oct 30 16:23:03.038172 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:03.038113 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:03.180599 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:03.180572 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:03.320649 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:03.320541 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:03.477640 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:03.477611 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:04.036822 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:04.036791 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:23:04.037206 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:04.036918 4861 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:23:04.038183 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:04.038166 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-58c95ddf47-6m2jw" Oct 30 16:23:04.125054 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:04.125021 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:04.262131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:04.262100 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:04.900824 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:04.900799 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-82w6j" Oct 30 16:23:05.830699 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:05.830671 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dfdc4b7f-lkk6g"] Oct 30 16:23:05.994054 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:05.994026 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:06.194605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:06.194528 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:06.333655 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:06.333619 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:06.621016 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:06.620979 4861 patch_prober.go:29] interesting pod/image-registry-5cc8d7fd6b-v2gt8 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 16:23:06.621210 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:06.621047 4861 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" podUID="fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:23:06.671349 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:06.671315 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:23:06.671488 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:06.671376 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:23:06.673795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:06.673769 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/21d827a7-6f18-4be7-86eb-b9141d12a44f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnmqt\" (UID: \"21d827a7-6f18-4be7-86eb-b9141d12a44f\") " pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:23:06.673903 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:06.673850 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/93b8a9b8-43df-4dc0-97da-4bfd11f373cd-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-gn2hp\" (UID: \"93b8a9b8-43df-4dc0-97da-4bfd11f373cd\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:23:06.699313 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:06.699291 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" Oct 30 16:23:06.708404 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:06.708388 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wnmqt" Oct 30 16:23:07.014127 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:07.014098 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp"] Oct 30 16:23:07.017562 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:23:07.017538 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b8a9b8_43df_4dc0_97da_4bfd11f373cd.slice/crio-7eac529eebc1d662234627802c3317f71444809f4db5bd5b159593778cd2a2ff WatchSource:0}: Error finding container 7eac529eebc1d662234627802c3317f71444809f4db5bd5b159593778cd2a2ff: Status 404 returned error can't find the container with id 7eac529eebc1d662234627802c3317f71444809f4db5bd5b159593778cd2a2ff Oct 30 16:23:07.020231 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:23:07.020206 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d827a7_6f18_4be7_86eb_b9141d12a44f.slice/crio-08f1d93d779aa7f13414ec15b6462f84b8a36b93520686d3a9ef9dbe7d6f1cfa WatchSource:0}: Error finding container 08f1d93d779aa7f13414ec15b6462f84b8a36b93520686d3a9ef9dbe7d6f1cfa: Status 404 returned error can't find the container with id 08f1d93d779aa7f13414ec15b6462f84b8a36b93520686d3a9ef9dbe7d6f1cfa Oct 30 16:23:07.026492 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:07.026472 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wnmqt"] Oct 30 16:23:07.123170 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:07.123144 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l95sm" Oct 30 16:23:07.956135 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:07.956103 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wnmqt" event={"ID":"21d827a7-6f18-4be7-86eb-b9141d12a44f","Type":"ContainerStarted","Data":"0224ccf23ccf1ee15fddd35201c73a06b60607698dd2be3916b7609c024a9069"} Oct 30 16:23:07.956255 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:07.956166 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wnmqt" event={"ID":"21d827a7-6f18-4be7-86eb-b9141d12a44f","Type":"ContainerStarted","Data":"df533913589d8b7ec7f5d8bf7e3913e16f6ead3a42868b8b931ef5cf3f02afa2"} Oct 30 16:23:07.956255 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:07.956194 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wnmqt" event={"ID":"21d827a7-6f18-4be7-86eb-b9141d12a44f","Type":"ContainerStarted","Data":"08f1d93d779aa7f13414ec15b6462f84b8a36b93520686d3a9ef9dbe7d6f1cfa"} Oct 30 16:23:07.957500 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:07.957473 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" event={"ID":"93b8a9b8-43df-4dc0-97da-4bfd11f373cd","Type":"ContainerStarted","Data":"7eac529eebc1d662234627802c3317f71444809f4db5bd5b159593778cd2a2ff"} Oct 30 16:23:08.964466 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:08.964383 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-gn2hp" event={"ID":"93b8a9b8-43df-4dc0-97da-4bfd11f373cd","Type":"ContainerStarted","Data":"127d8303499b1a802964698a8cfd8be5f987ca664002196732eb3824a279dca8"} Oct 30 16:23:08.996845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:08.996810 4861 patch_prober.go:29] interesting pod/image-registry-5cc8d7fd6b-v2gt8 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 16:23:08.997005 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:08.996870 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" podUID="fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:23:09.257509 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:09.257474 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:09.396562 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:09.396534 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:09.641965 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:09.641899 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:09.779824 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:09.779797 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:09.970011 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:09.969971 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wnmqt" event={"ID":"21d827a7-6f18-4be7-86eb-b9141d12a44f","Type":"ContainerStarted","Data":"0f26c6c1d2e1770baaf4c5d8c70a8d350be993c59a8cc9309bb8ad1412590c95"} Oct 30 16:23:12.727666 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:12.727602 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:12.870462 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:12.870430 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:13.020291 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:13.020224 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:14.588467 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:14.588434 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:14.729558 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:14.729527 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:14.941628 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:14.941561 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:15.084962 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:15.084931 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:15.645478 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:15.645439 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:15.836047 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:15.836023 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:23:17.189888 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.189846 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:17.332639 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.332611 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:17.473616 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.473580 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-g57lb"] Oct 30 16:23:17.504546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.504509 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.507822 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.507797 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:17.546156 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.546133 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97c994bf-ac26-438e-86c0-012c237ae965-node-exporter-accelerators-collector-config\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.546272 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.546163 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97c994bf-ac26-438e-86c0-012c237ae965-node-exporter-textfile\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.546272 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.546181 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97c994bf-ac26-438e-86c0-012c237ae965-node-exporter-wtmp\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.546272 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.546251 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97c994bf-ac26-438e-86c0-012c237ae965-node-exporter-tls\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.546372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.546287 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97c994bf-ac26-438e-86c0-012c237ae965-sys\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.546372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.546360 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97c994bf-ac26-438e-86c0-012c237ae965-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.546436 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.546387 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97c994bf-ac26-438e-86c0-012c237ae965-metrics-client-ca\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.546436 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.546410 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ss8t\" (UniqueName: \"kubernetes.io/projected/97c994bf-ac26-438e-86c0-012c237ae965-kube-api-access-7ss8t\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.546436 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.546426 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97c994bf-ac26-438e-86c0-012c237ae965-root\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.557933 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.557909 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 16:23:17.558042 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.557941 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-grp8r\"" Oct 30 16:23:17.558042 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.557994 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 16:23:17.570953 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.570920 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 16:23:17.571043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.570999 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 16:23:17.647548 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.647516 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ss8t\" (UniqueName: \"kubernetes.io/projected/97c994bf-ac26-438e-86c0-012c237ae965-kube-api-access-7ss8t\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.647693 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.647552 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97c994bf-ac26-438e-86c0-012c237ae965-root\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.647693 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.647620 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/97c994bf-ac26-438e-86c0-012c237ae965-root\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.647693 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.647656 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97c994bf-ac26-438e-86c0-012c237ae965-node-exporter-accelerators-collector-config\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.647693 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.647685 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97c994bf-ac26-438e-86c0-012c237ae965-node-exporter-textfile\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.647934 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.647712 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97c994bf-ac26-438e-86c0-012c237ae965-node-exporter-wtmp\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.647934 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.647766 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97c994bf-ac26-438e-86c0-012c237ae965-node-exporter-tls\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.647934 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.647798 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97c994bf-ac26-438e-86c0-012c237ae965-sys\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.647934 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.647843 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97c994bf-ac26-438e-86c0-012c237ae965-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.647934 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.647891 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97c994bf-ac26-438e-86c0-012c237ae965-sys\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.647934 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.647903 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97c994bf-ac26-438e-86c0-012c237ae965-metrics-client-ca\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.648227 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.647972 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/97c994bf-ac26-438e-86c0-012c237ae965-node-exporter-wtmp\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.648227 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.648126 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/97c994bf-ac26-438e-86c0-012c237ae965-node-exporter-textfile\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.648345 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.648260 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/97c994bf-ac26-438e-86c0-012c237ae965-node-exporter-accelerators-collector-config\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.648422 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.648403 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97c994bf-ac26-438e-86c0-012c237ae965-metrics-client-ca\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.650222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.650205 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/97c994bf-ac26-438e-86c0-012c237ae965-node-exporter-tls\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.650308 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.650262 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97c994bf-ac26-438e-86c0-012c237ae965-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.670668 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.670647 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ss8t\" (UniqueName: \"kubernetes.io/projected/97c994bf-ac26-438e-86c0-012c237ae965-kube-api-access-7ss8t\") pod \"node-exporter-g57lb\" (UID: \"97c994bf-ac26-438e-86c0-012c237ae965\") " pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.695884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.695855 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:17.848348 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.848272 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:17.852324 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.852300 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g57lb" Oct 30 16:23:17.882271 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:23:17.882247 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97c994bf_ac26_438e_86c0_012c237ae965.slice/crio-4709e021ca2af7159d0546c2aee7c4fe00db81508b754fd2048834942465bb34 WatchSource:0}: Error finding container 4709e021ca2af7159d0546c2aee7c4fe00db81508b754fd2048834942465bb34: Status 404 returned error can't find the container with id 4709e021ca2af7159d0546c2aee7c4fe00db81508b754fd2048834942465bb34 Oct 30 16:23:17.988595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.988568 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:17.998757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:17.998709 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g57lb" event={"ID":"97c994bf-ac26-438e-86c0-012c237ae965","Type":"ContainerStarted","Data":"4709e021ca2af7159d0546c2aee7c4fe00db81508b754fd2048834942465bb34"} Oct 30 16:23:18.561572 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:18.561528 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:18.698147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:18.698113 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:18.856340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:18.856260 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:18.900257 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:18.900231 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hptcz" Oct 30 16:23:18.996059 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:18.996035 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:23:19.131677 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:19.131607 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gjhv5" Oct 30 16:23:19.293577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:19.293529 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:19.466188 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:19.466115 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:20.007382 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:20.007346 4861 generic.go:356] "Generic (PLEG): container finished" podID="97c994bf-ac26-438e-86c0-012c237ae965" containerID="2627e60e7e3db2fba389533ee4d25d7f25a4d9a1e0637d13ddf013779c67b8fc" exitCode=0 Oct 30 16:23:20.007815 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:20.007407 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g57lb" event={"ID":"97c994bf-ac26-438e-86c0-012c237ae965","Type":"ContainerDied","Data":"2627e60e7e3db2fba389533ee4d25d7f25a4d9a1e0637d13ddf013779c67b8fc"} Oct 30 16:23:20.288150 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:23:20.288074 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:23:20.771629 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:20.771594 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:20.920782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:20.920742 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:21.013765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:21.013739 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g57lb" event={"ID":"97c994bf-ac26-438e-86c0-012c237ae965","Type":"ContainerStarted","Data":"a61f9e1911c1915ceb9b0cb6a4a6e3606bbf72c10bff3b456f10675e67a24014"} Oct 30 16:23:21.014139 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:21.013786 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g57lb" event={"ID":"97c994bf-ac26-438e-86c0-012c237ae965","Type":"ContainerStarted","Data":"37c647b97c433e4d1d23f130f2c6541b9f407bffd5bc58d06f377bc288c1fc9e"} Oct 30 16:23:21.058943 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:21.058836 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-g57lb" podStartSLOduration=2.713020369 podStartE2EDuration="4.058822823s" totalImagesPullingTime="1.259014024s" totalInitContainerRuntime="86.78843ms" isStatefulPod=true podCreationTimestamp="2026-10-30 16:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:23:21.057579584 +0000 UTC m=+31.400253632" watchObservedRunningTime="2026-10-30 16:23:21.058822823 +0000 UTC m=+31.401496855" Oct 30 16:23:22.254351 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:22.254298 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:22.461896 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:22.461838 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:22.612906 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:22.612826 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:24.011536 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:24.011504 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:24.246585 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:24.246526 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:24.445891 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:24.445803 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:28.127931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:28.127895 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8"] Oct 30 16:23:28.275266 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:28.275221 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:28.420604 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:28.420507 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:28.555405 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:28.555332 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:29.314943 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:29.314909 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:29.456854 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:29.456822 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:29.955392 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:29.955362 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:30.100622 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:30.100582 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:30.850767 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:30.850670 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" podUID="7932d7cc-b9fe-423f-bc88-93711eaec908" containerName="registry" containerID="cri-o://b9dc88d5b6e276503f81273536b8ff605b12ff8f2451b40d0d0cb302a6eb4156" gracePeriod=30 Oct 30 16:23:31.056016 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.055984 4861 generic.go:356] "Generic (PLEG): container finished" podID="7932d7cc-b9fe-423f-bc88-93711eaec908" containerID="b9dc88d5b6e276503f81273536b8ff605b12ff8f2451b40d0d0cb302a6eb4156" exitCode=0 Oct 30 16:23:31.056146 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.056051 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" event={"ID":"7932d7cc-b9fe-423f-bc88-93711eaec908","Type":"ContainerDied","Data":"b9dc88d5b6e276503f81273536b8ff605b12ff8f2451b40d0d0cb302a6eb4156"} Oct 30 16:23:31.101399 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.101354 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:23:31.156613 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.156590 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-installation-pull-secrets\") pod \"7932d7cc-b9fe-423f-bc88-93711eaec908\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " Oct 30 16:23:31.156751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.156643 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-trusted-ca\") pod \"7932d7cc-b9fe-423f-bc88-93711eaec908\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " Oct 30 16:23:31.156751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.156672 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-image-registry-private-configuration\") pod \"7932d7cc-b9fe-423f-bc88-93711eaec908\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " Oct 30 16:23:31.156751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.156704 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-kube-api-access-kghb4\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-kube-api-access-kghb4\") pod \"7932d7cc-b9fe-423f-bc88-93711eaec908\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " Oct 30 16:23:31.156920 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.156887 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-certificates\") pod \"7932d7cc-b9fe-423f-bc88-93711eaec908\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " Oct 30 16:23:31.156999 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.156979 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-bound-sa-token\") pod \"7932d7cc-b9fe-423f-bc88-93711eaec908\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " Oct 30 16:23:31.157057 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.157024 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls\") pod \"7932d7cc-b9fe-423f-bc88-93711eaec908\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " Oct 30 16:23:31.157120 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.157062 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7932d7cc-b9fe-423f-bc88-93711eaec908-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7932d7cc-b9fe-423f-bc88-93711eaec908-ca-trust-extracted\") pod \"7932d7cc-b9fe-423f-bc88-93711eaec908\" (UID: \"7932d7cc-b9fe-423f-bc88-93711eaec908\") " Oct 30 16:23:31.157192 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.157167 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-trusted-ca" pod "7932d7cc-b9fe-423f-bc88-93711eaec908" (UID: "7932d7cc-b9fe-423f-bc88-93711eaec908"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:23:31.157257 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.157233 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-certificates" pod "7932d7cc-b9fe-423f-bc88-93711eaec908" (UID: "7932d7cc-b9fe-423f-bc88-93711eaec908"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:23:31.157402 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.157384 4861 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-trusted-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:23:31.157629 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.157409 4861 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-certificates\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:23:31.159296 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.159270 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-installation-pull-secrets" pod "7932d7cc-b9fe-423f-bc88-93711eaec908" (UID: "7932d7cc-b9fe-423f-bc88-93711eaec908"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:23:31.159409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.159273 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-kube-api-access-kghb4" pod "7932d7cc-b9fe-423f-bc88-93711eaec908" (UID: "7932d7cc-b9fe-423f-bc88-93711eaec908"). InnerVolumeSpecName "kube-api-access-kghb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:23:31.159589 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.159558 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-image-registry-private-configuration" pod "7932d7cc-b9fe-423f-bc88-93711eaec908" (UID: "7932d7cc-b9fe-423f-bc88-93711eaec908"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:23:31.159760 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.159717 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-bound-sa-token" pod "7932d7cc-b9fe-423f-bc88-93711eaec908" (UID: "7932d7cc-b9fe-423f-bc88-93711eaec908"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:23:31.159760 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.159748 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls" pod "7932d7cc-b9fe-423f-bc88-93711eaec908" (UID: "7932d7cc-b9fe-423f-bc88-93711eaec908"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:23:31.165441 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.165397 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7932d7cc-b9fe-423f-bc88-93711eaec908-ca-trust-extracted" pod "7932d7cc-b9fe-423f-bc88-93711eaec908" (UID: "7932d7cc-b9fe-423f-bc88-93711eaec908"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:23:31.258131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.258111 4861 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-bound-sa-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:23:31.258131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.258132 4861 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-registry-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:23:31.258268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.258141 4861 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7932d7cc-b9fe-423f-bc88-93711eaec908-ca-trust-extracted\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:23:31.258268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.258149 4861 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-installation-pull-secrets\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:23:31.258268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.258159 4861 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7932d7cc-b9fe-423f-bc88-93711eaec908-image-registry-private-configuration\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:23:31.258268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:31.258168 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kghb4\" (UniqueName: \"kubernetes.io/projected/7932d7cc-b9fe-423f-bc88-93711eaec908-kube-api-access-kghb4\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:23:32.061360 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:32.061327 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" event={"ID":"7932d7cc-b9fe-423f-bc88-93711eaec908","Type":"ContainerDied","Data":"89aa7b4b39305f8d331379e508c24305d4c35ab972e67721f438e3a00b540292"} Oct 30 16:23:32.061360 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:32.061359 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dfdc4b7f-lkk6g" Oct 30 16:23:32.061836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:32.061387 4861 scope.go:122] "RemoveContainer" containerID="b9dc88d5b6e276503f81273536b8ff605b12ff8f2451b40d0d0cb302a6eb4156" Oct 30 16:23:32.129361 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:32.129336 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dfdc4b7f-lkk6g"] Oct 30 16:23:32.135570 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:32.135539 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5dfdc4b7f-lkk6g"] Oct 30 16:23:33.804068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:33.804036 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7932d7cc-b9fe-423f-bc88-93711eaec908" path="/var/lib/kubelet/pods/7932d7cc-b9fe-423f-bc88-93711eaec908/volumes" Oct 30 16:23:34.252950 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:34.252922 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:34.389027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:34.388965 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:34.748537 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:34.748506 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:34.887965 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:34.887935 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:35.944586 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:35.944552 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:36.091260 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:36.091230 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:39.729741 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:39.729706 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:39.965042 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:39.965006 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:44.936314 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:44.936253 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:45.085588 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:45.085555 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:23:50.288211 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:23:50.288174 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:23:53.154384 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.154324 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" podUID="fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" containerName="registry" containerID="cri-o://cea1272d07d14025d4d8bea0b59d2b86792c7751c6281122c326225f54f5d484" gracePeriod=30 Oct 30 16:23:53.401276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.401250 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:23:53.521465 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.521441 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-image-registry-private-configuration\") pod \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " Oct 30 16:23:53.521634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.521477 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-installation-pull-secrets\") pod \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " Oct 30 16:23:53.521634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.521502 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls\") pod \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " Oct 30 16:23:53.521634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.521527 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-trusted-ca\") pod \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " Oct 30 16:23:53.521836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.521652 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-ca-trust-extracted\") pod \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " Oct 30 16:23:53.521836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.521753 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-certificates\") pod \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " Oct 30 16:23:53.521836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.521779 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-bound-sa-token\") pod \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " Oct 30 16:23:53.521836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.521799 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-kube-api-access-zt2sw\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-kube-api-access-zt2sw\") pod \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\" (UID: \"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5\") " Oct 30 16:23:53.522095 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.522071 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-trusted-ca" pod "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" (UID: "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:23:53.522643 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.522594 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-certificates" pod "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" (UID: "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:23:53.524108 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.524082 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-image-registry-private-configuration" pod "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" (UID: "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:23:53.524472 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.524440 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-bound-sa-token" pod "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" (UID: "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:23:53.524554 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.524473 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-installation-pull-secrets" pod "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" (UID: "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:23:53.524716 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.524691 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls" pod "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" (UID: "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:23:53.524716 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.524703 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-kube-api-access-zt2sw" pod "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" (UID: "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5"). InnerVolumeSpecName "kube-api-access-zt2sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:23:53.528808 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.528789 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-ca-trust-extracted" pod "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" (UID: "fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:23:53.622664 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.622638 4861 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-image-registry-private-configuration\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:23:53.622664 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.622663 4861 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-installation-pull-secrets\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:23:53.622834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.622679 4861 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:23:53.622834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.622692 4861 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-trusted-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:23:53.622834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.622704 4861 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-ca-trust-extracted\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:23:53.622834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.622717 4861 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-registry-certificates\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:23:53.622834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.622756 4861 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-bound-sa-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:23:53.622834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:53.622769 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt2sw\" (UniqueName: \"kubernetes.io/projected/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5-kube-api-access-zt2sw\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:23:54.144634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:54.144557 4861 generic.go:356] "Generic (PLEG): container finished" podID="fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" containerID="cea1272d07d14025d4d8bea0b59d2b86792c7751c6281122c326225f54f5d484" exitCode=0 Oct 30 16:23:54.144634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:54.144609 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" event={"ID":"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5","Type":"ContainerDied","Data":"cea1272d07d14025d4d8bea0b59d2b86792c7751c6281122c326225f54f5d484"} Oct 30 16:23:54.144634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:54.144617 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" Oct 30 16:23:54.144895 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:54.144652 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8" event={"ID":"fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5","Type":"ContainerDied","Data":"ac5236464d8d1aef2df6c4a33522ca8940758c7cdb7918af7cc5a69048d9de1c"} Oct 30 16:23:54.144895 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:54.144673 4861 scope.go:122] "RemoveContainer" containerID="cea1272d07d14025d4d8bea0b59d2b86792c7751c6281122c326225f54f5d484" Oct 30 16:23:54.152649 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:54.152632 4861 scope.go:122] "RemoveContainer" containerID="cea1272d07d14025d4d8bea0b59d2b86792c7751c6281122c326225f54f5d484" Oct 30 16:23:54.152947 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:23:54.152927 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea1272d07d14025d4d8bea0b59d2b86792c7751c6281122c326225f54f5d484\": container with ID starting with cea1272d07d14025d4d8bea0b59d2b86792c7751c6281122c326225f54f5d484 not found: ID does not exist" containerID="cea1272d07d14025d4d8bea0b59d2b86792c7751c6281122c326225f54f5d484" Oct 30 16:23:54.152987 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:54.152954 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea1272d07d14025d4d8bea0b59d2b86792c7751c6281122c326225f54f5d484"} err="failed to get container status \"cea1272d07d14025d4d8bea0b59d2b86792c7751c6281122c326225f54f5d484\": rpc error: code = NotFound desc = could not find container \"cea1272d07d14025d4d8bea0b59d2b86792c7751c6281122c326225f54f5d484\": container with ID starting with cea1272d07d14025d4d8bea0b59d2b86792c7751c6281122c326225f54f5d484 not found: ID does not exist" Oct 30 16:23:54.209582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:54.209558 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8"] Oct 30 16:23:54.221822 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:54.221792 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5cc8d7fd6b-v2gt8"] Oct 30 16:23:55.804131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:23:55.804096 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" path="/var/lib/kubelet/pods/fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5/volumes" Oct 30 16:24:00.166058 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:00.165978 4861 generic.go:356] "Generic (PLEG): container finished" podID="a4922dc9-bd15-45a0-a9f6-e61089e6096b" containerID="6c4930a534499963dd70c355a8fb72f0d6cef3a539b72956ce22c8423fc5779c" exitCode=0 Oct 30 16:24:00.166397 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:00.166053 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" event={"ID":"a4922dc9-bd15-45a0-a9f6-e61089e6096b","Type":"ContainerDied","Data":"6c4930a534499963dd70c355a8fb72f0d6cef3a539b72956ce22c8423fc5779c"} Oct 30 16:24:00.166397 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:00.166385 4861 scope.go:122] "RemoveContainer" containerID="6c4930a534499963dd70c355a8fb72f0d6cef3a539b72956ce22c8423fc5779c" Oct 30 16:24:01.171478 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:01.171443 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p79fm" event={"ID":"a4922dc9-bd15-45a0-a9f6-e61089e6096b","Type":"ContainerStarted","Data":"38f9dc4729cdbf3ceaad073cf006b5e52bd79963915fbb9d53fb23538d46b2ec"} Oct 30 16:24:09.195903 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:09.195865 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:11.975975 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:11.975939 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:14.225004 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:14.224971 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:15.220224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:15.220192 4861 generic.go:356] "Generic (PLEG): container finished" podID="a1e1a5f5-7d62-43a9-9a51-1cb702e80963" containerID="c3fb517d5bc9c0c3d1152a3810d612436d51bbd2dd62ef8de8c9923927db7771" exitCode=0 Oct 30 16:24:15.220376 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:15.220267 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" event={"ID":"a1e1a5f5-7d62-43a9-9a51-1cb702e80963","Type":"ContainerDied","Data":"c3fb517d5bc9c0c3d1152a3810d612436d51bbd2dd62ef8de8c9923927db7771"} Oct 30 16:24:15.220639 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:15.220615 4861 scope.go:122] "RemoveContainer" containerID="c3fb517d5bc9c0c3d1152a3810d612436d51bbd2dd62ef8de8c9923927db7771" Oct 30 16:24:15.222100 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:15.222074 4861 generic.go:356] "Generic (PLEG): container finished" podID="9c282411-f8cb-4c83-8959-52035df7cd24" containerID="fc994fdddaae903e9e11783366592236bc35dd26c624858f80f5a77cb3594ba6" exitCode=0 Oct 30 16:24:15.222178 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:15.222114 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kx55q" event={"ID":"9c282411-f8cb-4c83-8959-52035df7cd24","Type":"ContainerDied","Data":"fc994fdddaae903e9e11783366592236bc35dd26c624858f80f5a77cb3594ba6"} Oct 30 16:24:15.222412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:15.222399 4861 scope.go:122] "RemoveContainer" containerID="fc994fdddaae903e9e11783366592236bc35dd26c624858f80f5a77cb3594ba6" Oct 30 16:24:15.933294 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:15.933264 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-kg6f8-gbh78"] Oct 30 16:24:15.933660 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:15.933533 4861 cpu_manager.go:414] "containerMap: removing container" podUID="fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" containerName="registry" Oct 30 16:24:15.933660 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:15.933544 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" containerName="registry" Oct 30 16:24:15.933660 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:15.933559 4861 cpu_manager.go:414] "containerMap: removing container" podUID="7932d7cc-b9fe-423f-bc88-93711eaec908" containerName="registry" Oct 30 16:24:15.933660 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:15.933564 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7932d7cc-b9fe-423f-bc88-93711eaec908" containerName="registry" Oct 30 16:24:15.933660 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:15.933613 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7c4e1f-a6c4-4303-8e1b-d9c043c62fc5" containerName="registry" Oct 30 16:24:15.933660 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:15.933622 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="7932d7cc-b9fe-423f-bc88-93711eaec908" containerName="registry" Oct 30 16:24:15.936524 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:15.936506 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kg6f8-gbh78" Oct 30 16:24:15.955455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:15.955431 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kg6f8-gbh78"] Oct 30 16:24:16.087557 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:16.087531 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a30de74-0d7c-48f3-a537-a53313fd7cde-trusted-ca-bundle\") pod \"periodic-gathering-kg6f8-gbh78\" (UID: \"2a30de74-0d7c-48f3-a537-a53313fd7cde\") " pod="openshift-insights/periodic-gathering-kg6f8-gbh78" Oct 30 16:24:16.087677 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:16.087583 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2a30de74-0d7c-48f3-a537-a53313fd7cde-archives-path\") pod \"periodic-gathering-kg6f8-gbh78\" (UID: \"2a30de74-0d7c-48f3-a537-a53313fd7cde\") " pod="openshift-insights/periodic-gathering-kg6f8-gbh78" Oct 30 16:24:16.087677 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:16.087651 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6jhq\" (UniqueName: \"kubernetes.io/projected/2a30de74-0d7c-48f3-a537-a53313fd7cde-kube-api-access-f6jhq\") pod \"periodic-gathering-kg6f8-gbh78\" (UID: \"2a30de74-0d7c-48f3-a537-a53313fd7cde\") " pod="openshift-insights/periodic-gathering-kg6f8-gbh78" Oct 30 16:24:16.087789 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:16.087679 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a30de74-0d7c-48f3-a537-a53313fd7cde-service-ca-bundle\") pod \"periodic-gathering-kg6f8-gbh78\" (UID: \"2a30de74-0d7c-48f3-a537-a53313fd7cde\") " pod="openshift-insights/periodic-gathering-kg6f8-gbh78" Oct 30 16:24:16.188806 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:16.188738 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2a30de74-0d7c-48f3-a537-a53313fd7cde-archives-path\") pod \"periodic-gathering-kg6f8-gbh78\" (UID: \"2a30de74-0d7c-48f3-a537-a53313fd7cde\") " pod="openshift-insights/periodic-gathering-kg6f8-gbh78" Oct 30 16:24:16.188806 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:16.188777 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6jhq\" (UniqueName: \"kubernetes.io/projected/2a30de74-0d7c-48f3-a537-a53313fd7cde-kube-api-access-f6jhq\") pod \"periodic-gathering-kg6f8-gbh78\" (UID: \"2a30de74-0d7c-48f3-a537-a53313fd7cde\") " pod="openshift-insights/periodic-gathering-kg6f8-gbh78" Oct 30 16:24:16.188806 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:16.188802 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a30de74-0d7c-48f3-a537-a53313fd7cde-service-ca-bundle\") pod \"periodic-gathering-kg6f8-gbh78\" (UID: \"2a30de74-0d7c-48f3-a537-a53313fd7cde\") " pod="openshift-insights/periodic-gathering-kg6f8-gbh78" Oct 30 16:24:16.189016 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:16.188859 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a30de74-0d7c-48f3-a537-a53313fd7cde-trusted-ca-bundle\") pod \"periodic-gathering-kg6f8-gbh78\" (UID: \"2a30de74-0d7c-48f3-a537-a53313fd7cde\") " pod="openshift-insights/periodic-gathering-kg6f8-gbh78" Oct 30 16:24:16.189102 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:16.189080 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2a30de74-0d7c-48f3-a537-a53313fd7cde-archives-path\") pod \"periodic-gathering-kg6f8-gbh78\" (UID: \"2a30de74-0d7c-48f3-a537-a53313fd7cde\") " pod="openshift-insights/periodic-gathering-kg6f8-gbh78" Oct 30 16:24:16.189430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:16.189407 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a30de74-0d7c-48f3-a537-a53313fd7cde-service-ca-bundle\") pod \"periodic-gathering-kg6f8-gbh78\" (UID: \"2a30de74-0d7c-48f3-a537-a53313fd7cde\") " pod="openshift-insights/periodic-gathering-kg6f8-gbh78" Oct 30 16:24:16.189680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:16.189660 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a30de74-0d7c-48f3-a537-a53313fd7cde-trusted-ca-bundle\") pod \"periodic-gathering-kg6f8-gbh78\" (UID: \"2a30de74-0d7c-48f3-a537-a53313fd7cde\") " pod="openshift-insights/periodic-gathering-kg6f8-gbh78" Oct 30 16:24:16.212577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:16.212544 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6jhq\" (UniqueName: \"kubernetes.io/projected/2a30de74-0d7c-48f3-a537-a53313fd7cde-kube-api-access-f6jhq\") pod \"periodic-gathering-kg6f8-gbh78\" (UID: \"2a30de74-0d7c-48f3-a537-a53313fd7cde\") " pod="openshift-insights/periodic-gathering-kg6f8-gbh78" Oct 30 16:24:16.227440 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:16.227409 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-kx55q" event={"ID":"9c282411-f8cb-4c83-8959-52035df7cd24","Type":"ContainerStarted","Data":"3d6db9458ab6c73f8b50184782958825b44e34e69cfa63043238c423bb346781"} Oct 30 16:24:16.229180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:16.229162 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-wgr44" event={"ID":"a1e1a5f5-7d62-43a9-9a51-1cb702e80963","Type":"ContainerStarted","Data":"1b50fb30d34cbdaa00c8891681b0e45a99dce2cbe9780b7a7c211adfbec932c1"} Oct 30 16:24:16.247832 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:16.247812 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kg6f8-gbh78" Oct 30 16:24:16.548570 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:16.548543 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kg6f8-gbh78"] Oct 30 16:24:16.551929 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:24:16.551900 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a30de74_0d7c_48f3_a537_a53313fd7cde.slice/crio-7fdfc4785538510b4e68b6532c76d4754859f1b810a0836953637d8e15557a12 WatchSource:0}: Error finding container 7fdfc4785538510b4e68b6532c76d4754859f1b810a0836953637d8e15557a12: Status 404 returned error can't find the container with id 7fdfc4785538510b4e68b6532c76d4754859f1b810a0836953637d8e15557a12 Oct 30 16:24:17.235056 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:17.235020 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kg6f8-gbh78" event={"ID":"2a30de74-0d7c-48f3-a537-a53313fd7cde","Type":"ContainerStarted","Data":"bc982dbb0ad04c2de2653a848430ed0a44fd24e08373f0a9866b2c06766938d3"} Oct 30 16:24:17.235437 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:17.235082 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kg6f8-gbh78" event={"ID":"2a30de74-0d7c-48f3-a537-a53313fd7cde","Type":"ContainerStarted","Data":"7fdfc4785538510b4e68b6532c76d4754859f1b810a0836953637d8e15557a12"} Oct 30 16:24:17.276776 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:17.276684 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-kg6f8-gbh78" podStartSLOduration=2.276671126 podStartE2EDuration="2.276671126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:24:17.275767475 +0000 UTC m=+87.618441513" watchObservedRunningTime="2026-10-30 16:24:17.276671126 +0000 UTC m=+87.619345163" Oct 30 16:24:20.287787 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:24:20.287742 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:24:24.803173 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:24.803135 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:24.946421 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:24.946386 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:29.934739 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:29.934678 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:30.078331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:30.078298 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:39.850564 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:39.850527 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:43.197437 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:43.197403 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:43.337461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:43.337419 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:43.491404 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:43.491371 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:43.628400 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:24:43.628368 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:24:50.287179 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:24:50.287139 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:25:02.196165 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:02.196061 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:02.330142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:02.330113 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:20.288179 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:25:20.288130 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:25:39.977142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:39.977111 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:40.532192 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:40.532158 4861 generic.go:356] "Generic (PLEG): container finished" podID="2a30de74-0d7c-48f3-a537-a53313fd7cde" containerID="bc982dbb0ad04c2de2653a848430ed0a44fd24e08373f0a9866b2c06766938d3" exitCode=0 Oct 30 16:25:40.532365 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:40.532206 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kg6f8-gbh78" event={"ID":"2a30de74-0d7c-48f3-a537-a53313fd7cde","Type":"ContainerDied","Data":"bc982dbb0ad04c2de2653a848430ed0a44fd24e08373f0a9866b2c06766938d3"} Oct 30 16:25:41.659467 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:41.659444 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kg6f8-gbh78" Oct 30 16:25:41.804256 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:41.804192 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a30de74-0d7c-48f3-a537-a53313fd7cde-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a30de74-0d7c-48f3-a537-a53313fd7cde-service-ca-bundle\") pod \"2a30de74-0d7c-48f3-a537-a53313fd7cde\" (UID: \"2a30de74-0d7c-48f3-a537-a53313fd7cde\") " Oct 30 16:25:41.804386 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:41.804260 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a30de74-0d7c-48f3-a537-a53313fd7cde-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a30de74-0d7c-48f3-a537-a53313fd7cde-trusted-ca-bundle\") pod \"2a30de74-0d7c-48f3-a537-a53313fd7cde\" (UID: \"2a30de74-0d7c-48f3-a537-a53313fd7cde\") " Oct 30 16:25:41.804386 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:41.804283 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a30de74-0d7c-48f3-a537-a53313fd7cde-kube-api-access-f6jhq\" (UniqueName: \"kubernetes.io/projected/2a30de74-0d7c-48f3-a537-a53313fd7cde-kube-api-access-f6jhq\") pod \"2a30de74-0d7c-48f3-a537-a53313fd7cde\" (UID: \"2a30de74-0d7c-48f3-a537-a53313fd7cde\") " Oct 30 16:25:41.804386 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:41.804329 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a30de74-0d7c-48f3-a537-a53313fd7cde-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2a30de74-0d7c-48f3-a537-a53313fd7cde-archives-path\") pod \"2a30de74-0d7c-48f3-a537-a53313fd7cde\" (UID: \"2a30de74-0d7c-48f3-a537-a53313fd7cde\") " Oct 30 16:25:41.804650 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:41.804622 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a30de74-0d7c-48f3-a537-a53313fd7cde-service-ca-bundle" pod "2a30de74-0d7c-48f3-a537-a53313fd7cde" (UID: "2a30de74-0d7c-48f3-a537-a53313fd7cde"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:25:41.804761 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:41.804629 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a30de74-0d7c-48f3-a537-a53313fd7cde-trusted-ca-bundle" pod "2a30de74-0d7c-48f3-a537-a53313fd7cde" (UID: "2a30de74-0d7c-48f3-a537-a53313fd7cde"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:25:41.804761 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:41.804672 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a30de74-0d7c-48f3-a537-a53313fd7cde-archives-path" pod "2a30de74-0d7c-48f3-a537-a53313fd7cde" (UID: "2a30de74-0d7c-48f3-a537-a53313fd7cde"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:25:41.806708 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:41.806687 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a30de74-0d7c-48f3-a537-a53313fd7cde-kube-api-access-f6jhq" pod "2a30de74-0d7c-48f3-a537-a53313fd7cde" (UID: "2a30de74-0d7c-48f3-a537-a53313fd7cde"). InnerVolumeSpecName "kube-api-access-f6jhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:25:41.905067 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:41.905044 4861 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2a30de74-0d7c-48f3-a537-a53313fd7cde-archives-path\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:25:41.905160 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:41.905069 4861 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a30de74-0d7c-48f3-a537-a53313fd7cde-service-ca-bundle\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:25:41.905219 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:41.905163 4861 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a30de74-0d7c-48f3-a537-a53313fd7cde-trusted-ca-bundle\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:25:41.905313 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:41.905294 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6jhq\" (UniqueName: \"kubernetes.io/projected/2a30de74-0d7c-48f3-a537-a53313fd7cde-kube-api-access-f6jhq\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:25:42.541862 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:42.541829 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kg6f8-gbh78" event={"ID":"2a30de74-0d7c-48f3-a537-a53313fd7cde","Type":"ContainerDied","Data":"7fdfc4785538510b4e68b6532c76d4754859f1b810a0836953637d8e15557a12"} Oct 30 16:25:42.542005 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:42.541877 4861 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fdfc4785538510b4e68b6532c76d4754859f1b810a0836953637d8e15557a12" Oct 30 16:25:42.542005 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:42.541843 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kg6f8-gbh78" Oct 30 16:25:43.067328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:43.067299 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:43.214241 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:25:43.214212 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:25:50.288801 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:25:50.288761 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:26:16.025377 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:16.025342 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:16.175088 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:16.175057 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:20.288861 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:26:20.288821 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:26:34.021950 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:34.021869 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:34.218063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:34.218028 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:35.109083 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.109053 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst"] Oct 30 16:26:35.109466 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.109335 4861 cpu_manager.go:414] "containerMap: removing container" podUID="2a30de74-0d7c-48f3-a537-a53313fd7cde" containerName="insights-gathering" Oct 30 16:26:35.109466 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.109345 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a30de74-0d7c-48f3-a537-a53313fd7cde" containerName="insights-gathering" Oct 30 16:26:35.109466 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.109403 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a30de74-0d7c-48f3-a537-a53313fd7cde" containerName="insights-gathering" Oct 30 16:26:35.112584 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.112554 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" Oct 30 16:26:35.113389 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.113368 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6"] Oct 30 16:26:35.116440 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.116425 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" Oct 30 16:26:35.130371 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.130351 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 16:26:35.130648 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.130633 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 16:26:35.130695 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.130675 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-cjsww\"" Oct 30 16:26:35.142833 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.142811 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-bhhng\"" Oct 30 16:26:35.144188 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.144153 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst"] Oct 30 16:26:35.145008 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.144987 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6"] Oct 30 16:26:35.157470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.157450 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62ee61b5-6cd7-4277-8301-011f8c45b443-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst\" (UID: \"62ee61b5-6cd7-4277-8301-011f8c45b443\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" Oct 30 16:26:35.157561 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.157477 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62ee61b5-6cd7-4277-8301-011f8c45b443-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst\" (UID: \"62ee61b5-6cd7-4277-8301-011f8c45b443\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" Oct 30 16:26:35.157561 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.157506 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc2vf\" (UniqueName: \"kubernetes.io/projected/62ee61b5-6cd7-4277-8301-011f8c45b443-kube-api-access-kc2vf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst\" (UID: \"62ee61b5-6cd7-4277-8301-011f8c45b443\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" Oct 30 16:26:35.157643 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.157591 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6\" (UID: \"2995e6fa-1f63-4672-99eb-8a10ac1cc88c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" Oct 30 16:26:35.157747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.157714 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxmzc\" (UniqueName: \"kubernetes.io/projected/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-kube-api-access-fxmzc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6\" (UID: \"2995e6fa-1f63-4672-99eb-8a10ac1cc88c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" Oct 30 16:26:35.157824 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.157811 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6\" (UID: \"2995e6fa-1f63-4672-99eb-8a10ac1cc88c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" Oct 30 16:26:35.258440 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.258411 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxmzc\" (UniqueName: \"kubernetes.io/projected/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-kube-api-access-fxmzc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6\" (UID: \"2995e6fa-1f63-4672-99eb-8a10ac1cc88c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" Oct 30 16:26:35.258582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.258457 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6\" (UID: \"2995e6fa-1f63-4672-99eb-8a10ac1cc88c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" Oct 30 16:26:35.258582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.258488 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62ee61b5-6cd7-4277-8301-011f8c45b443-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst\" (UID: \"62ee61b5-6cd7-4277-8301-011f8c45b443\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" Oct 30 16:26:35.258582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.258514 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62ee61b5-6cd7-4277-8301-011f8c45b443-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst\" (UID: \"62ee61b5-6cd7-4277-8301-011f8c45b443\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" Oct 30 16:26:35.258582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.258559 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc2vf\" (UniqueName: \"kubernetes.io/projected/62ee61b5-6cd7-4277-8301-011f8c45b443-kube-api-access-kc2vf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst\" (UID: \"62ee61b5-6cd7-4277-8301-011f8c45b443\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" Oct 30 16:26:35.258827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.258585 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6\" (UID: \"2995e6fa-1f63-4672-99eb-8a10ac1cc88c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" Oct 30 16:26:35.258998 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.258980 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62ee61b5-6cd7-4277-8301-011f8c45b443-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst\" (UID: \"62ee61b5-6cd7-4277-8301-011f8c45b443\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" Oct 30 16:26:35.259065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.259006 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6\" (UID: \"2995e6fa-1f63-4672-99eb-8a10ac1cc88c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" Oct 30 16:26:35.259065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.259026 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62ee61b5-6cd7-4277-8301-011f8c45b443-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst\" (UID: \"62ee61b5-6cd7-4277-8301-011f8c45b443\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" Oct 30 16:26:35.259065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.259028 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6\" (UID: \"2995e6fa-1f63-4672-99eb-8a10ac1cc88c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" Oct 30 16:26:35.282835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.282808 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxmzc\" (UniqueName: \"kubernetes.io/projected/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-kube-api-access-fxmzc\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6\" (UID: \"2995e6fa-1f63-4672-99eb-8a10ac1cc88c\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" Oct 30 16:26:35.283334 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.283318 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc2vf\" (UniqueName: \"kubernetes.io/projected/62ee61b5-6cd7-4277-8301-011f8c45b443-kube-api-access-kc2vf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst\" (UID: \"62ee61b5-6cd7-4277-8301-011f8c45b443\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" Oct 30 16:26:35.429887 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.429805 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" Oct 30 16:26:35.438069 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.438043 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" Oct 30 16:26:35.629097 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.629060 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb"] Oct 30 16:26:35.632451 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.632430 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.647814 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.647793 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 16:26:35.647911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.647895 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 16:26:35.648045 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.648033 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 16:26:35.648107 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.648046 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 16:26:35.648155 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.648114 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pmfs7\"" Oct 30 16:26:35.658356 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.658337 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb"] Oct 30 16:26:35.662147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.662124 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.662255 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.662164 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f361327-60e1-48a4-b4ff-b0da88163798-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.662340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.662319 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f361327-60e1-48a4-b4ff-b0da88163798-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.662399 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.662364 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f361327-60e1-48a4-b4ff-b0da88163798-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.662465 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.662443 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4vch\" (UniqueName: \"kubernetes.io/projected/2f361327-60e1-48a4-b4ff-b0da88163798-kube-api-access-s4vch\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.662506 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.662491 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.662590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.662571 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.665464 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.665444 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:35.747958 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.747927 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst"] Oct 30 16:26:35.750367 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:26:35.750348 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ee61b5_6cd7_4277_8301_011f8c45b443.slice/crio-d4980f70bac0ced4dc841da7ba6077960c8a863479a5d144692a602469f98bc9 WatchSource:0}: Error finding container d4980f70bac0ced4dc841da7ba6077960c8a863479a5d144692a602469f98bc9: Status 404 returned error can't find the container with id d4980f70bac0ced4dc841da7ba6077960c8a863479a5d144692a602469f98bc9 Oct 30 16:26:35.755835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.755746 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6"] Oct 30 16:26:35.758026 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:26:35.758006 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2995e6fa_1f63_4672_99eb_8a10ac1cc88c.slice/crio-fe18f17a2f6a83683ff875bdc370e0b6c952471c4cea207afa5fec49017e4812 WatchSource:0}: Error finding container fe18f17a2f6a83683ff875bdc370e0b6c952471c4cea207afa5fec49017e4812: Status 404 returned error can't find the container with id fe18f17a2f6a83683ff875bdc370e0b6c952471c4cea207afa5fec49017e4812 Oct 30 16:26:35.763633 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.763614 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f361327-60e1-48a4-b4ff-b0da88163798-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.763747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.763643 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f361327-60e1-48a4-b4ff-b0da88163798-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.763747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.763661 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4vch\" (UniqueName: \"kubernetes.io/projected/2f361327-60e1-48a4-b4ff-b0da88163798-kube-api-access-s4vch\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.763747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.763678 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.763747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.763708 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.763948 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.763766 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.763948 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.763793 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f361327-60e1-48a4-b4ff-b0da88163798-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.764375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.764354 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f361327-60e1-48a4-b4ff-b0da88163798-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.766323 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.766307 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.766506 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.766486 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f361327-60e1-48a4-b4ff-b0da88163798-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.766574 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.766559 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.766812 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.766799 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.786779 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.786755 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4vch\" (UniqueName: \"kubernetes.io/projected/2f361327-60e1-48a4-b4ff-b0da88163798-kube-api-access-s4vch\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.786858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.786796 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f361327-60e1-48a4-b4ff-b0da88163798-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mdnbb\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:35.827285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.827267 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:35.943798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:35.943768 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:36.264499 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:36.264471 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb"] Oct 30 16:26:36.268031 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:26:36.268005 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f361327_60e1_48a4_b4ff_b0da88163798.slice/crio-7402ae4d48a1702c02321303fe712bfe46c8f5f93461957c82e325f23772296a WatchSource:0}: Error finding container 7402ae4d48a1702c02321303fe712bfe46c8f5f93461957c82e325f23772296a: Status 404 returned error can't find the container with id 7402ae4d48a1702c02321303fe712bfe46c8f5f93461957c82e325f23772296a Oct 30 16:26:36.737970 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:36.737855 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" event={"ID":"2995e6fa-1f63-4672-99eb-8a10ac1cc88c","Type":"ContainerStarted","Data":"fe18f17a2f6a83683ff875bdc370e0b6c952471c4cea207afa5fec49017e4812"} Oct 30 16:26:36.739563 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:36.739480 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" event={"ID":"62ee61b5-6cd7-4277-8301-011f8c45b443","Type":"ContainerStarted","Data":"d4980f70bac0ced4dc841da7ba6077960c8a863479a5d144692a602469f98bc9"} Oct 30 16:26:36.741695 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:36.741660 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" event={"ID":"2f361327-60e1-48a4-b4ff-b0da88163798","Type":"ContainerStarted","Data":"7402ae4d48a1702c02321303fe712bfe46c8f5f93461957c82e325f23772296a"} Oct 30 16:26:37.247356 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:37.247319 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:39.387226 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:39.387186 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:26:39.387598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:39.387274 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:26:39.760032 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:39.759961 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" event={"ID":"2f361327-60e1-48a4-b4ff-b0da88163798","Type":"ContainerStarted","Data":"86c6744c804dec4da0c3d0f84d803268c82836147f3e2a5575bcc32b297c9b4a"} Oct 30 16:26:39.760296 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:39.760253 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:39.831368 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:39.831313 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" podStartSLOduration=1.715925757 podStartE2EDuration="4.831295596s" totalImagesPullingTime="3.115369839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:26:39.818531169 +0000 UTC m=+230.161205213" watchObservedRunningTime="2026-10-30 16:26:39.831295596 +0000 UTC m=+230.173969632" Oct 30 16:26:40.139227 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:40.139130 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:40.290119 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:40.290085 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:40.560290 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:40.560238 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:40.708443 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:40.708367 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:40.768575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:40.768334 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:26:42.775972 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:42.775944 4861 generic.go:356] "Generic (PLEG): container finished" podID="2995e6fa-1f63-4672-99eb-8a10ac1cc88c" containerID="e8a142b6bb54fe8ec45b647911a6bf7071d55ad4e325983ec9fd6431162edb5b" exitCode=0 Oct 30 16:26:42.776313 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:42.776023 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" event={"ID":"2995e6fa-1f63-4672-99eb-8a10ac1cc88c","Type":"ContainerDied","Data":"e8a142b6bb54fe8ec45b647911a6bf7071d55ad4e325983ec9fd6431162edb5b"} Oct 30 16:26:42.777560 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:42.777540 4861 generic.go:356] "Generic (PLEG): container finished" podID="62ee61b5-6cd7-4277-8301-011f8c45b443" containerID="0eacf0ff54c69ee52ab44d9eccb60965a53da3df820f8d830ee4c6ce76e8297f" exitCode=0 Oct 30 16:26:42.777650 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:42.777596 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" event={"ID":"62ee61b5-6cd7-4277-8301-011f8c45b443","Type":"ContainerDied","Data":"0eacf0ff54c69ee52ab44d9eccb60965a53da3df820f8d830ee4c6ce76e8297f"} Oct 30 16:26:44.089074 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:44.089037 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:45.792438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:45.792403 4861 generic.go:356] "Generic (PLEG): container finished" podID="2995e6fa-1f63-4672-99eb-8a10ac1cc88c" containerID="e66d2900ec55aa5e2ad351715e6dad63f010b433a1c96de460714b0c3b2bb199" exitCode=0 Oct 30 16:26:45.792853 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:45.792496 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" event={"ID":"2995e6fa-1f63-4672-99eb-8a10ac1cc88c","Type":"ContainerDied","Data":"e66d2900ec55aa5e2ad351715e6dad63f010b433a1c96de460714b0c3b2bb199"} Oct 30 16:26:45.794552 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:45.794524 4861 generic.go:356] "Generic (PLEG): container finished" podID="62ee61b5-6cd7-4277-8301-011f8c45b443" containerID="f0fd563a52b1401481096706f82eee103b5949908e7c02513b4feaba0953e855" exitCode=0 Oct 30 16:26:45.794635 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:45.794593 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" event={"ID":"62ee61b5-6cd7-4277-8301-011f8c45b443","Type":"ContainerDied","Data":"f0fd563a52b1401481096706f82eee103b5949908e7c02513b4feaba0953e855"} Oct 30 16:26:50.292596 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:26:50.292555 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:26:52.826032 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:52.825998 4861 generic.go:356] "Generic (PLEG): container finished" podID="2995e6fa-1f63-4672-99eb-8a10ac1cc88c" containerID="59e17afc0acc1b50e2dd20c12ed8697f42fecb8ce7318103e1a306ae948d41cd" exitCode=0 Oct 30 16:26:52.826396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:52.826078 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" event={"ID":"2995e6fa-1f63-4672-99eb-8a10ac1cc88c","Type":"ContainerDied","Data":"59e17afc0acc1b50e2dd20c12ed8697f42fecb8ce7318103e1a306ae948d41cd"} Oct 30 16:26:52.828258 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:52.828234 4861 generic.go:356] "Generic (PLEG): container finished" podID="62ee61b5-6cd7-4277-8301-011f8c45b443" containerID="4731f7a082a97eb5d2c004b22deb6cce4fcb7afc0888e64b60f20b1c2e9bdef5" exitCode=0 Oct 30 16:26:52.828378 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:52.828298 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" event={"ID":"62ee61b5-6cd7-4277-8301-011f8c45b443","Type":"ContainerDied","Data":"4731f7a082a97eb5d2c004b22deb6cce4fcb7afc0888e64b60f20b1c2e9bdef5"} Oct 30 16:26:53.983690 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:53.983663 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" Oct 30 16:26:53.986518 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:53.986499 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" Oct 30 16:26:54.127206 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.127133 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-kube-api-access-fxmzc\" (UniqueName: \"kubernetes.io/projected/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-kube-api-access-fxmzc\") pod \"2995e6fa-1f63-4672-99eb-8a10ac1cc88c\" (UID: \"2995e6fa-1f63-4672-99eb-8a10ac1cc88c\") " Oct 30 16:26:54.127358 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.127211 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-util\" (UniqueName: \"kubernetes.io/empty-dir/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-util\") pod \"2995e6fa-1f63-4672-99eb-8a10ac1cc88c\" (UID: \"2995e6fa-1f63-4672-99eb-8a10ac1cc88c\") " Oct 30 16:26:54.127358 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.127243 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ee61b5-6cd7-4277-8301-011f8c45b443-bundle\" (UniqueName: \"kubernetes.io/empty-dir/62ee61b5-6cd7-4277-8301-011f8c45b443-bundle\") pod \"62ee61b5-6cd7-4277-8301-011f8c45b443\" (UID: \"62ee61b5-6cd7-4277-8301-011f8c45b443\") " Oct 30 16:26:54.127358 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.127266 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ee61b5-6cd7-4277-8301-011f8c45b443-kube-api-access-kc2vf\" (UniqueName: \"kubernetes.io/projected/62ee61b5-6cd7-4277-8301-011f8c45b443-kube-api-access-kc2vf\") pod \"62ee61b5-6cd7-4277-8301-011f8c45b443\" (UID: \"62ee61b5-6cd7-4277-8301-011f8c45b443\") " Oct 30 16:26:54.127358 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.127293 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-bundle\") pod \"2995e6fa-1f63-4672-99eb-8a10ac1cc88c\" (UID: \"2995e6fa-1f63-4672-99eb-8a10ac1cc88c\") " Oct 30 16:26:54.127591 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.127398 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62ee61b5-6cd7-4277-8301-011f8c45b443-util\" (UniqueName: \"kubernetes.io/empty-dir/62ee61b5-6cd7-4277-8301-011f8c45b443-util\") pod \"62ee61b5-6cd7-4277-8301-011f8c45b443\" (UID: \"62ee61b5-6cd7-4277-8301-011f8c45b443\") " Oct 30 16:26:54.127773 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.127752 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-bundle" pod "2995e6fa-1f63-4672-99eb-8a10ac1cc88c" (UID: "2995e6fa-1f63-4672-99eb-8a10ac1cc88c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:26:54.128326 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.128299 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ee61b5-6cd7-4277-8301-011f8c45b443-bundle" pod "62ee61b5-6cd7-4277-8301-011f8c45b443" (UID: "62ee61b5-6cd7-4277-8301-011f8c45b443"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:26:54.129688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.129661 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ee61b5-6cd7-4277-8301-011f8c45b443-kube-api-access-kc2vf" pod "62ee61b5-6cd7-4277-8301-011f8c45b443" (UID: "62ee61b5-6cd7-4277-8301-011f8c45b443"). InnerVolumeSpecName "kube-api-access-kc2vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:26:54.129884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.129843 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-kube-api-access-fxmzc" pod "2995e6fa-1f63-4672-99eb-8a10ac1cc88c" (UID: "2995e6fa-1f63-4672-99eb-8a10ac1cc88c"). InnerVolumeSpecName "kube-api-access-fxmzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:26:54.131403 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.131375 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62ee61b5-6cd7-4277-8301-011f8c45b443-util" pod "62ee61b5-6cd7-4277-8301-011f8c45b443" (UID: "62ee61b5-6cd7-4277-8301-011f8c45b443"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:26:54.132194 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.132168 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-util" pod "2995e6fa-1f63-4672-99eb-8a10ac1cc88c" (UID: "2995e6fa-1f63-4672-99eb-8a10ac1cc88c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:26:54.228960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.228935 4861 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-util\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:26:54.228960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.228957 4861 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62ee61b5-6cd7-4277-8301-011f8c45b443-bundle\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:26:54.229077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.228967 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc2vf\" (UniqueName: \"kubernetes.io/projected/62ee61b5-6cd7-4277-8301-011f8c45b443-kube-api-access-kc2vf\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:26:54.229077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.228977 4861 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-bundle\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:26:54.229077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.228985 4861 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62ee61b5-6cd7-4277-8301-011f8c45b443-util\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:26:54.229077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.228993 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxmzc\" (UniqueName: \"kubernetes.io/projected/2995e6fa-1f63-4672-99eb-8a10ac1cc88c-kube-api-access-fxmzc\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:26:54.837211 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.837186 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" Oct 30 16:26:54.837396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.837177 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447hqp6" event={"ID":"2995e6fa-1f63-4672-99eb-8a10ac1cc88c","Type":"ContainerDied","Data":"fe18f17a2f6a83683ff875bdc370e0b6c952471c4cea207afa5fec49017e4812"} Oct 30 16:26:54.837396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.837305 4861 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe18f17a2f6a83683ff875bdc370e0b6c952471c4cea207afa5fec49017e4812" Oct 30 16:26:54.839257 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.839237 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" Oct 30 16:26:54.839381 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.839258 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czsqst" event={"ID":"62ee61b5-6cd7-4277-8301-011f8c45b443","Type":"ContainerDied","Data":"d4980f70bac0ced4dc841da7ba6077960c8a863479a5d144692a602469f98bc9"} Oct 30 16:26:54.839381 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:54.839298 4861 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4980f70bac0ced4dc841da7ba6077960c8a863479a5d144692a602469f98bc9" Oct 30 16:26:55.791812 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:55.791780 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:55.925217 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:55.925190 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:26:58.694659 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:26:58.694621 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:08.328481 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:08.328438 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:10.344100 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:10.344066 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:10.489224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:10.489196 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:10.737974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:10.737946 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:11.086889 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.086818 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:11.263032 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.263000 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:11.265008 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.264987 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs"] Oct 30 16:27:11.265290 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.265278 4861 cpu_manager.go:414] "containerMap: removing container" podUID="2995e6fa-1f63-4672-99eb-8a10ac1cc88c" containerName="extract" Oct 30 16:27:11.265328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.265291 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2995e6fa-1f63-4672-99eb-8a10ac1cc88c" containerName="extract" Oct 30 16:27:11.265328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.265306 4861 cpu_manager.go:414] "containerMap: removing container" podUID="62ee61b5-6cd7-4277-8301-011f8c45b443" containerName="extract" Oct 30 16:27:11.265328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.265311 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ee61b5-6cd7-4277-8301-011f8c45b443" containerName="extract" Oct 30 16:27:11.265328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.265320 4861 cpu_manager.go:414] "containerMap: removing container" podUID="62ee61b5-6cd7-4277-8301-011f8c45b443" containerName="util" Oct 30 16:27:11.265328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.265325 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ee61b5-6cd7-4277-8301-011f8c45b443" containerName="util" Oct 30 16:27:11.265470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.265338 4861 cpu_manager.go:414] "containerMap: removing container" podUID="2995e6fa-1f63-4672-99eb-8a10ac1cc88c" containerName="pull" Oct 30 16:27:11.265470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.265343 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2995e6fa-1f63-4672-99eb-8a10ac1cc88c" containerName="pull" Oct 30 16:27:11.265470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.265360 4861 cpu_manager.go:414] "containerMap: removing container" podUID="2995e6fa-1f63-4672-99eb-8a10ac1cc88c" containerName="util" Oct 30 16:27:11.265470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.265364 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2995e6fa-1f63-4672-99eb-8a10ac1cc88c" containerName="util" Oct 30 16:27:11.265470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.265370 4861 cpu_manager.go:414] "containerMap: removing container" podUID="62ee61b5-6cd7-4277-8301-011f8c45b443" containerName="pull" Oct 30 16:27:11.265470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.265374 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ee61b5-6cd7-4277-8301-011f8c45b443" containerName="pull" Oct 30 16:27:11.265470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.265416 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="2995e6fa-1f63-4672-99eb-8a10ac1cc88c" containerName="extract" Oct 30 16:27:11.265470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.265424 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ee61b5-6cd7-4277-8301-011f8c45b443" containerName="extract" Oct 30 16:27:11.268284 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.268262 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.284306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.284280 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs"] Oct 30 16:27:11.342479 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.342414 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 16:27:11.342590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.342524 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 16:27:11.342688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.342668 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 16:27:11.342777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.342746 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 16:27:11.342816 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.342789 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 16:27:11.342863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.342847 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jwzjl\"" Oct 30 16:27:11.348143 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.348123 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-manifests\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.348451 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.348157 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-metrics-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.348451 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.348178 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.348451 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.348213 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-tmp\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.348451 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.348308 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vntln\" (UniqueName: \"kubernetes.io/projected/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-kube-api-access-vntln\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.348451 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.348335 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.408321 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.408293 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:11.448989 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.448970 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vntln\" (UniqueName: \"kubernetes.io/projected/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-kube-api-access-vntln\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.449084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.448999 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.449084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.449035 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-manifests\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.449174 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.449152 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-metrics-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.449216 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.449200 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.449268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.449228 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-tmp\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.449353 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.449337 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-manifests\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.451527 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.451507 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-tmp\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.451780 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.451762 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.451884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.451865 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.451884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.451874 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-metrics-cert\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.472918 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.472898 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vntln\" (UniqueName: \"kubernetes.io/projected/0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d-kube-api-access-vntln\") pod \"opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs\" (UID: \"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.557831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.557776 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:11.583683 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.583658 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:11.702070 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.702035 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:11.846068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.845988 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:11.953485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:11.953463 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs"] Oct 30 16:27:11.955856 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:27:11.955830 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf5c2a4_1aa8_4a6a_9cd0_9baa48c57b7d.slice/crio-2661d5971ea33ed4f57876747ed79e7af662e986c9ec2fb94892b671f6cb17c4 WatchSource:0}: Error finding container 2661d5971ea33ed4f57876747ed79e7af662e986c9ec2fb94892b671f6cb17c4: Status 404 returned error can't find the container with id 2661d5971ea33ed4f57876747ed79e7af662e986c9ec2fb94892b671f6cb17c4 Oct 30 16:27:12.912880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:12.912839 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" event={"ID":"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d","Type":"ContainerStarted","Data":"2661d5971ea33ed4f57876747ed79e7af662e986c9ec2fb94892b671f6cb17c4"} Oct 30 16:27:13.758035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:13.757974 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:13.895032 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:13.895005 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:14.922945 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:14.922898 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" event={"ID":"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d","Type":"ContainerStarted","Data":"8855d2819c83e100ad768ae6b88c8d823e3670cb2c94bd1c9f6ab6b414ba5251"} Oct 30 16:27:15.929404 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:15.929371 4861 generic.go:356] "Generic (PLEG): container finished" podID="0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d" containerID="8855d2819c83e100ad768ae6b88c8d823e3670cb2c94bd1c9f6ab6b414ba5251" exitCode=0 Oct 30 16:27:15.929798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:15.929430 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" event={"ID":"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d","Type":"ContainerDied","Data":"8855d2819c83e100ad768ae6b88c8d823e3670cb2c94bd1c9f6ab6b414ba5251"} Oct 30 16:27:16.937227 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:16.937178 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" event={"ID":"0cf5c2a4-1aa8-4a6a-9cd0-9baa48c57b7d","Type":"ContainerStarted","Data":"77f3b07b568b343421c90b4a0572296cad9d3e0e2d3eac935b259f798545a7eb"} Oct 30 16:27:16.937709 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:16.937464 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:17.026456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:17.026404 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" podStartSLOduration=2.339324422 podStartE2EDuration="6.026389496s" totalImagesPullingTime="3.277959159s" totalInitContainerRuntime="409.105915ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:17.013176877 +0000 UTC m=+267.355850915" watchObservedRunningTime="2026-10-30 16:27:17.026389496 +0000 UTC m=+267.369063533" Oct 30 16:27:17.351882 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:17.351847 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:17.492565 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:17.492529 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:19.515156 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:19.515117 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:20.287450 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:27:20.287404 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:27:20.731727 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:20.731703 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:21.720177 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:21.720148 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:26.126758 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:26.126689 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:27.943221 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:27.943191 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6cbf9dc6c8-c4zzs" Oct 30 16:27:28.551688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:28.551657 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:31.550605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:31.550571 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:31.690848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:31.690816 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:31.922196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:31.922071 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:32.154821 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:32.154769 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:32.759918 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:32.759881 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:32.915587 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:32.915539 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:33.982551 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:33.982519 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:34.149957 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:34.149925 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:35.033284 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:35.033246 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:35.246415 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:35.246388 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:35.407401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:35.407326 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:35.686058 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:35.685967 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:35.846298 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:35.846270 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:37.154674 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:37.154647 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:37.305000 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:37.300909 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:37.517213 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:37.517179 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:37.658118 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:37.658087 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:38.174693 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:38.174657 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:38.329685 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:38.329655 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:40.205673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:40.205640 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:40.351698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:40.351663 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:43.863256 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:43.863228 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:44.015043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:44.015013 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:45.292067 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:45.292040 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:45.924850 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:45.924822 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:46.089732 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.089688 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:46.246604 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.246579 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:46.515353 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.515268 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:46.659680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.659651 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:46.729365 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.729334 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk"] Oct 30 16:27:46.732788 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.732772 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk" Oct 30 16:27:46.752962 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.752936 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk"] Oct 30 16:27:46.763627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.763605 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 16:27:46.763812 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.763653 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 16:27:46.763812 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.763670 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-p6cx2\"" Oct 30 16:27:46.816427 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.816328 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:46.825860 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.825831 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac6c05a9-6522-473f-b77f-556378d2aa00-tmp\") pod \"kube-auth-proxy-b54f85fcc-p7zbk\" (UID: \"ac6c05a9-6522-473f-b77f-556378d2aa00\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk" Oct 30 16:27:46.825996 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.825876 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gq7z\" (UniqueName: \"kubernetes.io/projected/ac6c05a9-6522-473f-b77f-556378d2aa00-kube-api-access-8gq7z\") pod \"kube-auth-proxy-b54f85fcc-p7zbk\" (UID: \"ac6c05a9-6522-473f-b77f-556378d2aa00\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk" Oct 30 16:27:46.825996 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.825977 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6c05a9-6522-473f-b77f-556378d2aa00-tls-certs\") pod \"kube-auth-proxy-b54f85fcc-p7zbk\" (UID: \"ac6c05a9-6522-473f-b77f-556378d2aa00\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk" Oct 30 16:27:46.926534 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.926506 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac6c05a9-6522-473f-b77f-556378d2aa00-tmp\") pod \"kube-auth-proxy-b54f85fcc-p7zbk\" (UID: \"ac6c05a9-6522-473f-b77f-556378d2aa00\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk" Oct 30 16:27:46.926715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.926566 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gq7z\" (UniqueName: \"kubernetes.io/projected/ac6c05a9-6522-473f-b77f-556378d2aa00-kube-api-access-8gq7z\") pod \"kube-auth-proxy-b54f85fcc-p7zbk\" (UID: \"ac6c05a9-6522-473f-b77f-556378d2aa00\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk" Oct 30 16:27:46.926715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.926619 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6c05a9-6522-473f-b77f-556378d2aa00-tls-certs\") pod \"kube-auth-proxy-b54f85fcc-p7zbk\" (UID: \"ac6c05a9-6522-473f-b77f-556378d2aa00\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk" Oct 30 16:27:46.929236 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.929214 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6c05a9-6522-473f-b77f-556378d2aa00-tls-certs\") pod \"kube-auth-proxy-b54f85fcc-p7zbk\" (UID: \"ac6c05a9-6522-473f-b77f-556378d2aa00\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk" Oct 30 16:27:46.929341 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.929254 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac6c05a9-6522-473f-b77f-556378d2aa00-tmp\") pod \"kube-auth-proxy-b54f85fcc-p7zbk\" (UID: \"ac6c05a9-6522-473f-b77f-556378d2aa00\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk" Oct 30 16:27:46.957619 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.957587 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gq7z\" (UniqueName: \"kubernetes.io/projected/ac6c05a9-6522-473f-b77f-556378d2aa00-kube-api-access-8gq7z\") pod \"kube-auth-proxy-b54f85fcc-p7zbk\" (UID: \"ac6c05a9-6522-473f-b77f-556378d2aa00\") " pod="openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk" Oct 30 16:27:46.980204 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:46.980173 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:47.044988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:47.044963 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk" Oct 30 16:27:47.124788 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:47.124759 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:47.269769 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:47.269734 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:47.355944 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:47.355876 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk"] Oct 30 16:27:47.359239 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:27:47.359214 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6c05a9_6522_473f_b77f_556378d2aa00.slice/crio-ffe99a05d2a9204e97d6ff9fb958c8fb3ecc2d5a1c33bb0ba5e6d86c68b3a1a1 WatchSource:0}: Error finding container ffe99a05d2a9204e97d6ff9fb958c8fb3ecc2d5a1c33bb0ba5e6d86c68b3a1a1: Status 404 returned error can't find the container with id ffe99a05d2a9204e97d6ff9fb958c8fb3ecc2d5a1c33bb0ba5e6d86c68b3a1a1 Oct 30 16:27:47.409293 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:47.409266 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:48.066249 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:48.066199 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk" event={"ID":"ac6c05a9-6522-473f-b77f-556378d2aa00","Type":"ContainerStarted","Data":"ffe99a05d2a9204e97d6ff9fb958c8fb3ecc2d5a1c33bb0ba5e6d86c68b3a1a1"} Oct 30 16:27:48.535267 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:48.535233 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:49.465193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:49.465164 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:49.825207 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:49.825183 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:27:49.825369 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:49.825187 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:27:49.857746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:49.857707 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 16:27:50.076468 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:50.076387 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk" event={"ID":"ac6c05a9-6522-473f-b77f-556378d2aa00","Type":"ContainerStarted","Data":"15e1db822428f852d62145fd151910fe67daa944247f7dc8db539f75ea399fb2"} Oct 30 16:27:50.117564 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:50.117515 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b54f85fcc-p7zbk" podStartSLOduration=1.6351840850000001 podStartE2EDuration="4.117501428s" totalImagesPullingTime="2.482317343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:50.116892713 +0000 UTC m=+300.459566751" watchObservedRunningTime="2026-10-30 16:27:50.117501428 +0000 UTC m=+300.460175467" Oct 30 16:27:50.287794 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:27:50.287761 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:27:50.986684 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:50.986655 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6"] Oct 30 16:27:50.990242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:50.990220 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" Oct 30 16:27:51.005566 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.005548 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 16:27:51.017572 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.017552 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 16:27:51.043558 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.043534 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 16:27:51.043684 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.043661 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-mv9ss\"" Oct 30 16:27:51.044905 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.044890 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6"] Oct 30 16:27:51.056232 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.056205 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h9z8\" (UniqueName: \"kubernetes.io/projected/c07163d1-a743-4cc0-9dbf-4b1f7809e578-kube-api-access-8h9z8\") pod \"kserve-module-controller-manager-7566cf5fc8-5bzh6\" (UID: \"c07163d1-a743-4cc0-9dbf-4b1f7809e578\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" Oct 30 16:27:51.056285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.056251 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c07163d1-a743-4cc0-9dbf-4b1f7809e578-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-5bzh6\" (UID: \"c07163d1-a743-4cc0-9dbf-4b1f7809e578\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" Oct 30 16:27:51.056332 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.056319 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c07163d1-a743-4cc0-9dbf-4b1f7809e578-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-5bzh6\" (UID: \"c07163d1-a743-4cc0-9dbf-4b1f7809e578\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" Oct 30 16:27:51.056367 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.056341 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c07163d1-a743-4cc0-9dbf-4b1f7809e578-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-5bzh6\" (UID: \"c07163d1-a743-4cc0-9dbf-4b1f7809e578\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" Oct 30 16:27:51.157465 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.157439 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h9z8\" (UniqueName: \"kubernetes.io/projected/c07163d1-a743-4cc0-9dbf-4b1f7809e578-kube-api-access-8h9z8\") pod \"kserve-module-controller-manager-7566cf5fc8-5bzh6\" (UID: \"c07163d1-a743-4cc0-9dbf-4b1f7809e578\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" Oct 30 16:27:51.157576 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.157485 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c07163d1-a743-4cc0-9dbf-4b1f7809e578-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-5bzh6\" (UID: \"c07163d1-a743-4cc0-9dbf-4b1f7809e578\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" Oct 30 16:27:51.157654 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.157634 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c07163d1-a743-4cc0-9dbf-4b1f7809e578-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-5bzh6\" (UID: \"c07163d1-a743-4cc0-9dbf-4b1f7809e578\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" Oct 30 16:27:51.157701 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.157677 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c07163d1-a743-4cc0-9dbf-4b1f7809e578-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-5bzh6\" (UID: \"c07163d1-a743-4cc0-9dbf-4b1f7809e578\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" Oct 30 16:27:51.157985 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.157967 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c07163d1-a743-4cc0-9dbf-4b1f7809e578-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-5bzh6\" (UID: \"c07163d1-a743-4cc0-9dbf-4b1f7809e578\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" Oct 30 16:27:51.158053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.158003 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c07163d1-a743-4cc0-9dbf-4b1f7809e578-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-5bzh6\" (UID: \"c07163d1-a743-4cc0-9dbf-4b1f7809e578\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" Oct 30 16:27:51.158297 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.158279 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c07163d1-a743-4cc0-9dbf-4b1f7809e578-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-5bzh6\" (UID: \"c07163d1-a743-4cc0-9dbf-4b1f7809e578\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" Oct 30 16:27:51.181220 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.181197 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h9z8\" (UniqueName: \"kubernetes.io/projected/c07163d1-a743-4cc0-9dbf-4b1f7809e578-kube-api-access-8h9z8\") pod \"kserve-module-controller-manager-7566cf5fc8-5bzh6\" (UID: \"c07163d1-a743-4cc0-9dbf-4b1f7809e578\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" Oct 30 16:27:51.302761 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.302683 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" Oct 30 16:27:51.617922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:51.617849 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6"] Oct 30 16:27:51.620694 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:27:51.620664 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07163d1_a743_4cc0_9dbf_4b1f7809e578.slice/crio-0de0c4f985754345d96a4e8bfcd023abe052da68c1651446d8c721568bf40974 WatchSource:0}: Error finding container 0de0c4f985754345d96a4e8bfcd023abe052da68c1651446d8c721568bf40974: Status 404 returned error can't find the container with id 0de0c4f985754345d96a4e8bfcd023abe052da68c1651446d8c721568bf40974 Oct 30 16:27:52.086956 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:52.086908 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" event={"ID":"c07163d1-a743-4cc0-9dbf-4b1f7809e578","Type":"ContainerStarted","Data":"0de0c4f985754345d96a4e8bfcd023abe052da68c1651446d8c721568bf40974"} Oct 30 16:27:53.091840 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:53.091807 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" event={"ID":"c07163d1-a743-4cc0-9dbf-4b1f7809e578","Type":"ContainerStarted","Data":"57f873e3cf7346201e808a864bbd1e88597a055cadf9e4574d3f8c18c1144762"} Oct 30 16:27:53.092168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:53.091916 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" Oct 30 16:27:53.151160 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:53.151116 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" podStartSLOduration=1.810702897 podStartE2EDuration="3.151103022s" totalImagesPullingTime="1.340400125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:27:53.149700954 +0000 UTC m=+303.492374993" watchObservedRunningTime="2026-10-30 16:27:53.151103022 +0000 UTC m=+303.493777057" Oct 30 16:27:55.343541 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:55.343505 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:55.484581 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:55.484551 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:55.682786 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:55.682681 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:59.151335 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:59.151296 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:27:59.288048 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:27:59.288013 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:01.609487 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:01.609406 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:01.742279 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:01.742248 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:02.745202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:02.745169 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:02.878899 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:02.878868 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:14.097444 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:14.097416 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-5bzh6" Oct 30 16:28:15.850376 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:15.850346 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:15.995198 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:15.995158 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:16.649749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:16.649678 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:16.782114 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:16.782083 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:20.288093 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:28:20.288057 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:28:29.084021 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:29.083989 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:29.243526 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:29.243488 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:34.358499 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:34.358465 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:34.503506 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:34.503473 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:35.060340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.060306 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-tsg47"] Oct 30 16:28:35.063387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.063366 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-tsg47" Oct 30 16:28:35.082203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.082148 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:28:35.093875 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.093851 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:28:35.094186 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.094163 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-psqgt\"" Oct 30 16:28:35.094784 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.094764 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 16:28:35.095483 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.095449 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-tsg47"] Oct 30 16:28:35.184904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.184872 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:35.209044 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.209015 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ac065598-af1a-45c8-8720-00b2c12e4170-data\") pod \"postgres-7cdbf6dc44-tsg47\" (UID: \"ac065598-af1a-45c8-8720-00b2c12e4170\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-tsg47" Oct 30 16:28:35.209207 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.209052 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76pjn\" (UniqueName: \"kubernetes.io/projected/ac065598-af1a-45c8-8720-00b2c12e4170-kube-api-access-76pjn\") pod \"postgres-7cdbf6dc44-tsg47\" (UID: \"ac065598-af1a-45c8-8720-00b2c12e4170\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-tsg47" Oct 30 16:28:35.310245 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.310215 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ac065598-af1a-45c8-8720-00b2c12e4170-data\") pod \"postgres-7cdbf6dc44-tsg47\" (UID: \"ac065598-af1a-45c8-8720-00b2c12e4170\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-tsg47" Oct 30 16:28:35.310446 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.310269 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76pjn\" (UniqueName: \"kubernetes.io/projected/ac065598-af1a-45c8-8720-00b2c12e4170-kube-api-access-76pjn\") pod \"postgres-7cdbf6dc44-tsg47\" (UID: \"ac065598-af1a-45c8-8720-00b2c12e4170\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-tsg47" Oct 30 16:28:35.310640 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.310618 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ac065598-af1a-45c8-8720-00b2c12e4170-data\") pod \"postgres-7cdbf6dc44-tsg47\" (UID: \"ac065598-af1a-45c8-8720-00b2c12e4170\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-tsg47" Oct 30 16:28:35.334188 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.334157 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76pjn\" (UniqueName: \"kubernetes.io/projected/ac065598-af1a-45c8-8720-00b2c12e4170-kube-api-access-76pjn\") pod \"postgres-7cdbf6dc44-tsg47\" (UID: \"ac065598-af1a-45c8-8720-00b2c12e4170\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-tsg47" Oct 30 16:28:35.375603 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.375578 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-tsg47" Oct 30 16:28:35.674984 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.674897 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-tsg47"] Oct 30 16:28:35.678112 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:28:35.678083 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac065598_af1a_45c8_8720_00b2c12e4170.slice/crio-f3947a434271d4987032d3517b546b58d77c8ea52b06f41accfaf27d9a070913 WatchSource:0}: Error finding container f3947a434271d4987032d3517b546b58d77c8ea52b06f41accfaf27d9a070913: Status 404 returned error can't find the container with id f3947a434271d4987032d3517b546b58d77c8ea52b06f41accfaf27d9a070913 Oct 30 16:28:35.679296 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:35.679274 4861 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:28:36.278227 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:36.278195 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-tsg47" event={"ID":"ac065598-af1a-45c8-8720-00b2c12e4170","Type":"ContainerStarted","Data":"f3947a434271d4987032d3517b546b58d77c8ea52b06f41accfaf27d9a070913"} Oct 30 16:28:40.460053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:40.460023 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:41.309212 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:41.309172 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-tsg47" event={"ID":"ac065598-af1a-45c8-8720-00b2c12e4170","Type":"ContainerStarted","Data":"a205f064f6f452a40a9a8304437aa6d8c3e360194c8cd3be316b7cdb8db34bd4"} Oct 30 16:28:41.309384 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:41.309311 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-tsg47" Oct 30 16:28:41.353118 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:41.353071 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-tsg47" podStartSLOduration=1.604048949 podStartE2EDuration="6.353059136s" totalImagesPullingTime="4.749010187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:41.35241595 +0000 UTC m=+351.695089985" watchObservedRunningTime="2026-10-30 16:28:41.353059136 +0000 UTC m=+351.695733173" Oct 30 16:28:45.543846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.543815 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb"] Oct 30 16:28:45.547972 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.547952 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.576819 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.576789 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb"] Oct 30 16:28:45.597088 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.597055 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c2cf11eb-a585-4fdf-bfad-b6e70b866976-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.597088 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.597088 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c2cf11eb-a585-4fdf-bfad-b6e70b866976-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.597252 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.597162 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2cf11eb-a585-4fdf-bfad-b6e70b866976-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.597252 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.597183 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp7xd\" (UniqueName: \"kubernetes.io/projected/c2cf11eb-a585-4fdf-bfad-b6e70b866976-kube-api-access-wp7xd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.597252 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.597199 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2cf11eb-a585-4fdf-bfad-b6e70b866976-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.597379 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.597328 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2cf11eb-a585-4fdf-bfad-b6e70b866976-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.597436 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.597387 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c2cf11eb-a585-4fdf-bfad-b6e70b866976-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.698256 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.698229 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c2cf11eb-a585-4fdf-bfad-b6e70b866976-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.698387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.698283 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c2cf11eb-a585-4fdf-bfad-b6e70b866976-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.698387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.698303 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c2cf11eb-a585-4fdf-bfad-b6e70b866976-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.698387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.698332 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2cf11eb-a585-4fdf-bfad-b6e70b866976-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.698387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.698348 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp7xd\" (UniqueName: \"kubernetes.io/projected/c2cf11eb-a585-4fdf-bfad-b6e70b866976-kube-api-access-wp7xd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.698387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.698366 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2cf11eb-a585-4fdf-bfad-b6e70b866976-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.698639 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.698398 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2cf11eb-a585-4fdf-bfad-b6e70b866976-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.699100 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.699068 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c2cf11eb-a585-4fdf-bfad-b6e70b866976-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.700876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.700851 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2cf11eb-a585-4fdf-bfad-b6e70b866976-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.701034 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.700976 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c2cf11eb-a585-4fdf-bfad-b6e70b866976-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.701469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.701447 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c2cf11eb-a585-4fdf-bfad-b6e70b866976-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.701594 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.701573 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c2cf11eb-a585-4fdf-bfad-b6e70b866976-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.722398 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.722373 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp7xd\" (UniqueName: \"kubernetes.io/projected/c2cf11eb-a585-4fdf-bfad-b6e70b866976-kube-api-access-wp7xd\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.722535 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.722516 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2cf11eb-a585-4fdf-bfad-b6e70b866976-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-ssrxb\" (UID: \"c2cf11eb-a585-4fdf-bfad-b6e70b866976\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:45.858919 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:45.858854 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:46.248011 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:46.247989 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb"] Oct 30 16:28:46.249630 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:28:46.249596 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2cf11eb_a585_4fdf_bfad_b6e70b866976.slice/crio-ec43163274be70904f7276d1f9abfb3355df09d6fd95786bbdfb353aaea31f52 WatchSource:0}: Error finding container ec43163274be70904f7276d1f9abfb3355df09d6fd95786bbdfb353aaea31f52: Status 404 returned error can't find the container with id ec43163274be70904f7276d1f9abfb3355df09d6fd95786bbdfb353aaea31f52 Oct 30 16:28:46.251682 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:46.251655 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:28:46.251782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:46.251746 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:28:46.333793 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:46.333760 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" event={"ID":"c2cf11eb-a585-4fdf-bfad-b6e70b866976","Type":"ContainerStarted","Data":"d254f2ac3bba6ffe2e1a4aed957f35e49692065abf653569ed473a165b674b31"} Oct 30 16:28:46.333936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:46.333807 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" event={"ID":"c2cf11eb-a585-4fdf-bfad-b6e70b866976","Type":"ContainerStarted","Data":"ec43163274be70904f7276d1f9abfb3355df09d6fd95786bbdfb353aaea31f52"} Oct 30 16:28:46.333998 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:46.333981 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:46.381838 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:46.381788 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" podStartSLOduration=1.381774328 podStartE2EDuration="1.381774328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:46.380421926 +0000 UTC m=+356.723095964" watchObservedRunningTime="2026-10-30 16:28:46.381774328 +0000 UTC m=+356.724448365" Oct 30 16:28:46.536157 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:46.536081 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:46.686236 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:46.686198 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:46.834514 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:46.834421 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:47.008143 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.008106 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:47.165927 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.165857 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:47.340422 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.340398 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-ssrxb" Oct 30 16:28:47.343205 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.343175 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-tsg47" Oct 30 16:28:47.402461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.402428 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:47.423178 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.423112 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb"] Oct 30 16:28:47.423338 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.423317 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" podUID="2f361327-60e1-48a4-b4ff-b0da88163798" containerName="discovery" containerID="cri-o://86c6744c804dec4da0c3d0f84d803268c82836147f3e2a5575bcc32b297c9b4a" gracePeriod=30 Oct 30 16:28:47.667683 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.667660 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:28:47.718510 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.718456 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-istio-csr-dns-cert\") pod \"2f361327-60e1-48a4-b4ff-b0da88163798\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " Oct 30 16:28:47.719049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.718513 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-cacerts\" (UniqueName: \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-cacerts\") pod \"2f361327-60e1-48a4-b4ff-b0da88163798\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " Oct 30 16:28:47.719049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.718552 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f361327-60e1-48a4-b4ff-b0da88163798-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f361327-60e1-48a4-b4ff-b0da88163798-istio-csr-ca-configmap\") pod \"2f361327-60e1-48a4-b4ff-b0da88163798\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " Oct 30 16:28:47.719049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.718586 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f361327-60e1-48a4-b4ff-b0da88163798-istio-token\" (UniqueName: \"kubernetes.io/projected/2f361327-60e1-48a4-b4ff-b0da88163798-istio-token\") pod \"2f361327-60e1-48a4-b4ff-b0da88163798\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " Oct 30 16:28:47.719049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.718624 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-istio-kubeconfig\") pod \"2f361327-60e1-48a4-b4ff-b0da88163798\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " Oct 30 16:28:47.719049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.718698 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f361327-60e1-48a4-b4ff-b0da88163798-kube-api-access-s4vch\" (UniqueName: \"kubernetes.io/projected/2f361327-60e1-48a4-b4ff-b0da88163798-kube-api-access-s4vch\") pod \"2f361327-60e1-48a4-b4ff-b0da88163798\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " Oct 30 16:28:47.719049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.718756 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f361327-60e1-48a4-b4ff-b0da88163798-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f361327-60e1-48a4-b4ff-b0da88163798-local-certs\") pod \"2f361327-60e1-48a4-b4ff-b0da88163798\" (UID: \"2f361327-60e1-48a4-b4ff-b0da88163798\") " Oct 30 16:28:47.720985 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.720892 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f361327-60e1-48a4-b4ff-b0da88163798-istio-csr-ca-configmap" pod "2f361327-60e1-48a4-b4ff-b0da88163798" (UID: "2f361327-60e1-48a4-b4ff-b0da88163798"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:28:47.723227 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.723183 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f361327-60e1-48a4-b4ff-b0da88163798-local-certs" pod "2f361327-60e1-48a4-b4ff-b0da88163798" (UID: "2f361327-60e1-48a4-b4ff-b0da88163798"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:28:47.723430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.723402 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-cacerts" pod "2f361327-60e1-48a4-b4ff-b0da88163798" (UID: "2f361327-60e1-48a4-b4ff-b0da88163798"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:28:47.723602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.723562 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f361327-60e1-48a4-b4ff-b0da88163798-istio-token" pod "2f361327-60e1-48a4-b4ff-b0da88163798" (UID: "2f361327-60e1-48a4-b4ff-b0da88163798"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:28:47.723855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.723832 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-istio-csr-dns-cert" pod "2f361327-60e1-48a4-b4ff-b0da88163798" (UID: "2f361327-60e1-48a4-b4ff-b0da88163798"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:28:47.723945 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.723854 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-istio-kubeconfig" pod "2f361327-60e1-48a4-b4ff-b0da88163798" (UID: "2f361327-60e1-48a4-b4ff-b0da88163798"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:28:47.724121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.724095 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f361327-60e1-48a4-b4ff-b0da88163798-kube-api-access-s4vch" pod "2f361327-60e1-48a4-b4ff-b0da88163798" (UID: "2f361327-60e1-48a4-b4ff-b0da88163798"). InnerVolumeSpecName "kube-api-access-s4vch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:28:47.820132 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.820110 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4vch\" (UniqueName: \"kubernetes.io/projected/2f361327-60e1-48a4-b4ff-b0da88163798-kube-api-access-s4vch\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:28:47.820132 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.820134 4861 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f361327-60e1-48a4-b4ff-b0da88163798-local-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:28:47.820309 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.820151 4861 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-istio-csr-dns-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:28:47.820309 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.820172 4861 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-cacerts\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:28:47.820309 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.820258 4861 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f361327-60e1-48a4-b4ff-b0da88163798-istio-csr-ca-configmap\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:28:47.820309 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.820275 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f361327-60e1-48a4-b4ff-b0da88163798-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:28:47.820309 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:47.820288 4861 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f361327-60e1-48a4-b4ff-b0da88163798-istio-kubeconfig\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:28:48.344188 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:48.344159 4861 generic.go:356] "Generic (PLEG): container finished" podID="2f361327-60e1-48a4-b4ff-b0da88163798" containerID="86c6744c804dec4da0c3d0f84d803268c82836147f3e2a5575bcc32b297c9b4a" exitCode=0 Oct 30 16:28:48.344330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:48.344208 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" Oct 30 16:28:48.344330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:48.344240 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" event={"ID":"2f361327-60e1-48a4-b4ff-b0da88163798","Type":"ContainerDied","Data":"86c6744c804dec4da0c3d0f84d803268c82836147f3e2a5575bcc32b297c9b4a"} Oct 30 16:28:48.344330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:48.344302 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb" event={"ID":"2f361327-60e1-48a4-b4ff-b0da88163798","Type":"ContainerDied","Data":"7402ae4d48a1702c02321303fe712bfe46c8f5f93461957c82e325f23772296a"} Oct 30 16:28:48.344462 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:48.344326 4861 scope.go:122] "RemoveContainer" containerID="86c6744c804dec4da0c3d0f84d803268c82836147f3e2a5575bcc32b297c9b4a" Oct 30 16:28:48.352493 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:48.352475 4861 scope.go:122] "RemoveContainer" containerID="86c6744c804dec4da0c3d0f84d803268c82836147f3e2a5575bcc32b297c9b4a" Oct 30 16:28:48.352774 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:28:48.352754 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c6744c804dec4da0c3d0f84d803268c82836147f3e2a5575bcc32b297c9b4a\": container with ID starting with 86c6744c804dec4da0c3d0f84d803268c82836147f3e2a5575bcc32b297c9b4a not found: ID does not exist" containerID="86c6744c804dec4da0c3d0f84d803268c82836147f3e2a5575bcc32b297c9b4a" Oct 30 16:28:48.352828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:48.352786 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c6744c804dec4da0c3d0f84d803268c82836147f3e2a5575bcc32b297c9b4a"} err="failed to get container status \"86c6744c804dec4da0c3d0f84d803268c82836147f3e2a5575bcc32b297c9b4a\": rpc error: code = NotFound desc = could not find container \"86c6744c804dec4da0c3d0f84d803268c82836147f3e2a5575bcc32b297c9b4a\": container with ID starting with 86c6744c804dec4da0c3d0f84d803268c82836147f3e2a5575bcc32b297c9b4a not found: ID does not exist" Oct 30 16:28:48.396389 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:48.396364 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb"] Oct 30 16:28:48.411060 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:48.410947 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdnbb"] Oct 30 16:28:49.413312 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:49.413266 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:49.561995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:49.561963 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:49.701121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:49.701030 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:49.804291 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:49.804260 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f361327-60e1-48a4-b4ff-b0da88163798" path="/var/lib/kubelet/pods/2f361327-60e1-48a4-b4ff-b0da88163798/volumes" Oct 30 16:28:49.838561 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:49.838541 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:50.288011 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:28:50.287981 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:28:54.508307 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.508276 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:54.558560 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.558528 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5656d77c94-s4jnw"] Oct 30 16:28:54.558936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.558905 4861 cpu_manager.go:414] "containerMap: removing container" podUID="2f361327-60e1-48a4-b4ff-b0da88163798" containerName="discovery" Oct 30 16:28:54.558936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.558930 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f361327-60e1-48a4-b4ff-b0da88163798" containerName="discovery" Oct 30 16:28:54.559101 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.559047 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f361327-60e1-48a4-b4ff-b0da88163798" containerName="discovery" Oct 30 16:28:54.581753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.581714 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5656d77c94-s4jnw" Oct 30 16:28:54.587441 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.587410 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5656d77c94-s4jnw"] Oct 30 16:28:54.597634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.597607 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 16:28:54.598240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.598214 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 16:28:54.599258 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.599236 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 16:28:54.609456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.609428 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bp6hr\"" Oct 30 16:28:54.647244 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.647218 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:54.671513 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.671490 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ef20d61-4e5c-48e8-947c-2429fcbfe059-metrics-certs\") pod \"maas-controller-5656d77c94-s4jnw\" (UID: \"8ef20d61-4e5c-48e8-947c-2429fcbfe059\") " pod="opendatahub/maas-controller-5656d77c94-s4jnw" Oct 30 16:28:54.671609 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.671539 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ef20d61-4e5c-48e8-947c-2429fcbfe059-webhook-cert\") pod \"maas-controller-5656d77c94-s4jnw\" (UID: \"8ef20d61-4e5c-48e8-947c-2429fcbfe059\") " pod="opendatahub/maas-controller-5656d77c94-s4jnw" Oct 30 16:28:54.671609 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.671556 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cwv7\" (UniqueName: \"kubernetes.io/projected/8ef20d61-4e5c-48e8-947c-2429fcbfe059-kube-api-access-2cwv7\") pod \"maas-controller-5656d77c94-s4jnw\" (UID: \"8ef20d61-4e5c-48e8-947c-2429fcbfe059\") " pod="opendatahub/maas-controller-5656d77c94-s4jnw" Oct 30 16:28:54.772574 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.772498 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ef20d61-4e5c-48e8-947c-2429fcbfe059-metrics-certs\") pod \"maas-controller-5656d77c94-s4jnw\" (UID: \"8ef20d61-4e5c-48e8-947c-2429fcbfe059\") " pod="opendatahub/maas-controller-5656d77c94-s4jnw" Oct 30 16:28:54.772690 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.772576 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ef20d61-4e5c-48e8-947c-2429fcbfe059-webhook-cert\") pod \"maas-controller-5656d77c94-s4jnw\" (UID: \"8ef20d61-4e5c-48e8-947c-2429fcbfe059\") " pod="opendatahub/maas-controller-5656d77c94-s4jnw" Oct 30 16:28:54.772690 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.772602 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cwv7\" (UniqueName: \"kubernetes.io/projected/8ef20d61-4e5c-48e8-947c-2429fcbfe059-kube-api-access-2cwv7\") pod \"maas-controller-5656d77c94-s4jnw\" (UID: \"8ef20d61-4e5c-48e8-947c-2429fcbfe059\") " pod="opendatahub/maas-controller-5656d77c94-s4jnw" Oct 30 16:28:54.775121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.775096 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ef20d61-4e5c-48e8-947c-2429fcbfe059-metrics-certs\") pod \"maas-controller-5656d77c94-s4jnw\" (UID: \"8ef20d61-4e5c-48e8-947c-2429fcbfe059\") " pod="opendatahub/maas-controller-5656d77c94-s4jnw" Oct 30 16:28:54.775121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.775113 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ef20d61-4e5c-48e8-947c-2429fcbfe059-webhook-cert\") pod \"maas-controller-5656d77c94-s4jnw\" (UID: \"8ef20d61-4e5c-48e8-947c-2429fcbfe059\") " pod="opendatahub/maas-controller-5656d77c94-s4jnw" Oct 30 16:28:54.789077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.789052 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:54.795846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.795826 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cwv7\" (UniqueName: \"kubernetes.io/projected/8ef20d61-4e5c-48e8-947c-2429fcbfe059-kube-api-access-2cwv7\") pod \"maas-controller-5656d77c94-s4jnw\" (UID: \"8ef20d61-4e5c-48e8-947c-2429fcbfe059\") " pod="opendatahub/maas-controller-5656d77c94-s4jnw" Oct 30 16:28:54.911415 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.911387 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5656d77c94-s4jnw" Oct 30 16:28:54.932323 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:54.932298 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:55.082558 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:55.082475 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6475ccc8c4-q7cpf"] Oct 30 16:28:55.087150 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:55.087129 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6475ccc8c4-q7cpf" Oct 30 16:28:55.107498 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:55.107478 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6475ccc8c4-q7cpf"] Oct 30 16:28:55.176204 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:55.176179 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/33810159-2c87-4168-8c6d-6dec7af9a368-webhook-cert\") pod \"maas-controller-6475ccc8c4-q7cpf\" (UID: \"33810159-2c87-4168-8c6d-6dec7af9a368\") " pod="opendatahub/maas-controller-6475ccc8c4-q7cpf" Oct 30 16:28:55.176333 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:55.176223 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj96d\" (UniqueName: \"kubernetes.io/projected/33810159-2c87-4168-8c6d-6dec7af9a368-kube-api-access-jj96d\") pod \"maas-controller-6475ccc8c4-q7cpf\" (UID: \"33810159-2c87-4168-8c6d-6dec7af9a368\") " pod="opendatahub/maas-controller-6475ccc8c4-q7cpf" Oct 30 16:28:55.176333 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:55.176242 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33810159-2c87-4168-8c6d-6dec7af9a368-metrics-certs\") pod \"maas-controller-6475ccc8c4-q7cpf\" (UID: \"33810159-2c87-4168-8c6d-6dec7af9a368\") " pod="opendatahub/maas-controller-6475ccc8c4-q7cpf" Oct 30 16:28:55.213868 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:55.213848 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5656d77c94-s4jnw"] Oct 30 16:28:55.216446 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:28:55.216417 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef20d61_4e5c_48e8_947c_2429fcbfe059.slice/crio-83d60b587b04ebdca5872dd78f79e35f4d20f45d6c69402406d417a904588b6e WatchSource:0}: Error finding container 83d60b587b04ebdca5872dd78f79e35f4d20f45d6c69402406d417a904588b6e: Status 404 returned error can't find the container with id 83d60b587b04ebdca5872dd78f79e35f4d20f45d6c69402406d417a904588b6e Oct 30 16:28:55.277299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:55.277277 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj96d\" (UniqueName: \"kubernetes.io/projected/33810159-2c87-4168-8c6d-6dec7af9a368-kube-api-access-jj96d\") pod \"maas-controller-6475ccc8c4-q7cpf\" (UID: \"33810159-2c87-4168-8c6d-6dec7af9a368\") " pod="opendatahub/maas-controller-6475ccc8c4-q7cpf" Oct 30 16:28:55.277402 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:55.277309 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33810159-2c87-4168-8c6d-6dec7af9a368-metrics-certs\") pod \"maas-controller-6475ccc8c4-q7cpf\" (UID: \"33810159-2c87-4168-8c6d-6dec7af9a368\") " pod="opendatahub/maas-controller-6475ccc8c4-q7cpf" Oct 30 16:28:55.277402 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:55.277364 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/33810159-2c87-4168-8c6d-6dec7af9a368-webhook-cert\") pod \"maas-controller-6475ccc8c4-q7cpf\" (UID: \"33810159-2c87-4168-8c6d-6dec7af9a368\") " pod="opendatahub/maas-controller-6475ccc8c4-q7cpf" Oct 30 16:28:55.279844 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:55.279823 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/33810159-2c87-4168-8c6d-6dec7af9a368-webhook-cert\") pod \"maas-controller-6475ccc8c4-q7cpf\" (UID: \"33810159-2c87-4168-8c6d-6dec7af9a368\") " pod="opendatahub/maas-controller-6475ccc8c4-q7cpf" Oct 30 16:28:55.279924 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:55.279849 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33810159-2c87-4168-8c6d-6dec7af9a368-metrics-certs\") pod \"maas-controller-6475ccc8c4-q7cpf\" (UID: \"33810159-2c87-4168-8c6d-6dec7af9a368\") " pod="opendatahub/maas-controller-6475ccc8c4-q7cpf" Oct 30 16:28:55.301299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:55.301277 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj96d\" (UniqueName: \"kubernetes.io/projected/33810159-2c87-4168-8c6d-6dec7af9a368-kube-api-access-jj96d\") pod \"maas-controller-6475ccc8c4-q7cpf\" (UID: \"33810159-2c87-4168-8c6d-6dec7af9a368\") " pod="opendatahub/maas-controller-6475ccc8c4-q7cpf" Oct 30 16:28:55.380297 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:55.380232 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5656d77c94-s4jnw" event={"ID":"8ef20d61-4e5c-48e8-947c-2429fcbfe059","Type":"ContainerStarted","Data":"83d60b587b04ebdca5872dd78f79e35f4d20f45d6c69402406d417a904588b6e"} Oct 30 16:28:55.400027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:55.400013 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6475ccc8c4-q7cpf" Oct 30 16:28:55.702447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:55.702373 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6475ccc8c4-q7cpf"] Oct 30 16:28:56.391314 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:56.391269 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6475ccc8c4-q7cpf" event={"ID":"33810159-2c87-4168-8c6d-6dec7af9a368","Type":"ContainerStarted","Data":"71194747e02e563d3749f467df3e49b7c1b2222b9488f20153a6c9b0054e39e8"} Oct 30 16:28:58.190757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:58.190665 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:58.355050 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:58.355014 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:28:58.401531 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:58.401496 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6475ccc8c4-q7cpf" event={"ID":"33810159-2c87-4168-8c6d-6dec7af9a368","Type":"ContainerStarted","Data":"f2fbd143503fdfb297524e8d27777dd581e7133c03a544926be7cd9de252c5ad"} Oct 30 16:28:58.401701 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:58.401559 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6475ccc8c4-q7cpf" Oct 30 16:28:58.402958 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:58.402934 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5656d77c94-s4jnw" event={"ID":"8ef20d61-4e5c-48e8-947c-2429fcbfe059","Type":"ContainerStarted","Data":"3a2dbb28daf785f1c19679088752948f10485d1dc069adfabcc1dc5ef7630231"} Oct 30 16:28:58.403080 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:58.403064 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5656d77c94-s4jnw" Oct 30 16:28:58.472482 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:58.472437 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6475ccc8c4-q7cpf" podStartSLOduration=0.890138723 podStartE2EDuration="3.472421624s" totalImagesPullingTime="2.582282901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:58.471953896 +0000 UTC m=+368.814627957" watchObservedRunningTime="2026-10-30 16:28:58.472421624 +0000 UTC m=+368.815095663" Oct 30 16:28:58.535803 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:28:58.535765 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5656d77c94-s4jnw" podStartSLOduration=1.828624841 podStartE2EDuration="4.535752439s" totalImagesPullingTime="2.707127598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:58.535612087 +0000 UTC m=+368.878286127" watchObservedRunningTime="2026-10-30 16:28:58.535752439 +0000 UTC m=+368.878426474" Oct 30 16:29:00.248887 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.248856 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74"] Oct 30 16:29:00.252281 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.252261 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" Oct 30 16:29:00.276431 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.276399 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 16:29:00.288413 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.288393 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-prjzn\"" Oct 30 16:29:00.288571 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.288535 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 16:29:00.288571 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.288554 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 16:29:00.290085 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.290068 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74"] Oct 30 16:29:00.325710 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.325682 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9qlm\" (UniqueName: \"kubernetes.io/projected/be3b019c-c736-4857-bda5-92df56b650c5-kube-api-access-d9qlm\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-klj74\" (UID: \"be3b019c-c736-4857-bda5-92df56b650c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" Oct 30 16:29:00.325847 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.325754 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/be3b019c-c736-4857-bda5-92df56b650c5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-klj74\" (UID: \"be3b019c-c736-4857-bda5-92df56b650c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" Oct 30 16:29:00.325847 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.325817 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be3b019c-c736-4857-bda5-92df56b650c5-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-klj74\" (UID: \"be3b019c-c736-4857-bda5-92df56b650c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" Oct 30 16:29:00.325847 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.325834 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/be3b019c-c736-4857-bda5-92df56b650c5-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-klj74\" (UID: \"be3b019c-c736-4857-bda5-92df56b650c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" Oct 30 16:29:00.368842 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.368812 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:00.426897 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.426844 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9qlm\" (UniqueName: \"kubernetes.io/projected/be3b019c-c736-4857-bda5-92df56b650c5-kube-api-access-d9qlm\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-klj74\" (UID: \"be3b019c-c736-4857-bda5-92df56b650c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" Oct 30 16:29:00.426897 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.426898 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/be3b019c-c736-4857-bda5-92df56b650c5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-klj74\" (UID: \"be3b019c-c736-4857-bda5-92df56b650c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" Oct 30 16:29:00.427171 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.427104 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be3b019c-c736-4857-bda5-92df56b650c5-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-klj74\" (UID: \"be3b019c-c736-4857-bda5-92df56b650c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" Oct 30 16:29:00.427171 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.427124 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/be3b019c-c736-4857-bda5-92df56b650c5-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-klj74\" (UID: \"be3b019c-c736-4857-bda5-92df56b650c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" Oct 30 16:29:00.427781 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.427759 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be3b019c-c736-4857-bda5-92df56b650c5-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-klj74\" (UID: \"be3b019c-c736-4857-bda5-92df56b650c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" Oct 30 16:29:00.428107 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.428090 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/be3b019c-c736-4857-bda5-92df56b650c5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-klj74\" (UID: \"be3b019c-c736-4857-bda5-92df56b650c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" Oct 30 16:29:00.429897 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.429867 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/be3b019c-c736-4857-bda5-92df56b650c5-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-klj74\" (UID: \"be3b019c-c736-4857-bda5-92df56b650c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" Oct 30 16:29:00.452021 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.451999 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9qlm\" (UniqueName: \"kubernetes.io/projected/be3b019c-c736-4857-bda5-92df56b650c5-kube-api-access-d9qlm\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-klj74\" (UID: \"be3b019c-c736-4857-bda5-92df56b650c5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" Oct 30 16:29:00.528040 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.527983 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:00.565889 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.565867 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" Oct 30 16:29:00.866883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:00.866813 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74"] Oct 30 16:29:00.869551 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:29:00.869526 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3b019c_c736_4857_bda5_92df56b650c5.slice/crio-859fd05a12f7a60c0666aae1c2779c21c00d509153753a0e7d9e4b6171168389 WatchSource:0}: Error finding container 859fd05a12f7a60c0666aae1c2779c21c00d509153753a0e7d9e4b6171168389: Status 404 returned error can't find the container with id 859fd05a12f7a60c0666aae1c2779c21c00d509153753a0e7d9e4b6171168389 Oct 30 16:29:01.418543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:01.418506 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" event={"ID":"be3b019c-c736-4857-bda5-92df56b650c5","Type":"ContainerStarted","Data":"859fd05a12f7a60c0666aae1c2779c21c00d509153753a0e7d9e4b6171168389"} Oct 30 16:29:05.063145 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:05.063109 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:09.410291 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:09.410261 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5656d77c94-s4jnw" Oct 30 16:29:09.411232 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:09.411209 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6475ccc8c4-q7cpf" Oct 30 16:29:09.533128 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:09.533095 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5656d77c94-s4jnw"] Oct 30 16:29:09.533623 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:09.533431 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5656d77c94-s4jnw" podUID="8ef20d61-4e5c-48e8-947c-2429fcbfe059" containerName="manager" containerID="cri-o://3a2dbb28daf785f1c19679088752948f10485d1dc069adfabcc1dc5ef7630231" gracePeriod=30 Oct 30 16:29:09.784097 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:09.784069 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5656d77c94-s4jnw" Oct 30 16:29:09.904639 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:09.904601 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ef20d61-4e5c-48e8-947c-2429fcbfe059-webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ef20d61-4e5c-48e8-947c-2429fcbfe059-webhook-cert\") pod \"8ef20d61-4e5c-48e8-947c-2429fcbfe059\" (UID: \"8ef20d61-4e5c-48e8-947c-2429fcbfe059\") " Oct 30 16:29:09.904843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:09.904655 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ef20d61-4e5c-48e8-947c-2429fcbfe059-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ef20d61-4e5c-48e8-947c-2429fcbfe059-metrics-certs\") pod \"8ef20d61-4e5c-48e8-947c-2429fcbfe059\" (UID: \"8ef20d61-4e5c-48e8-947c-2429fcbfe059\") " Oct 30 16:29:09.904843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:09.904685 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef20d61-4e5c-48e8-947c-2429fcbfe059-kube-api-access-2cwv7\" (UniqueName: \"kubernetes.io/projected/8ef20d61-4e5c-48e8-947c-2429fcbfe059-kube-api-access-2cwv7\") pod \"8ef20d61-4e5c-48e8-947c-2429fcbfe059\" (UID: \"8ef20d61-4e5c-48e8-947c-2429fcbfe059\") " Oct 30 16:29:09.907442 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:09.907412 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef20d61-4e5c-48e8-947c-2429fcbfe059-kube-api-access-2cwv7" pod "8ef20d61-4e5c-48e8-947c-2429fcbfe059" (UID: "8ef20d61-4e5c-48e8-947c-2429fcbfe059"). InnerVolumeSpecName "kube-api-access-2cwv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:29:09.907442 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:09.907412 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef20d61-4e5c-48e8-947c-2429fcbfe059-webhook-cert" pod "8ef20d61-4e5c-48e8-947c-2429fcbfe059" (UID: "8ef20d61-4e5c-48e8-947c-2429fcbfe059"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:29:09.907585 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:09.907477 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef20d61-4e5c-48e8-947c-2429fcbfe059-metrics-certs" pod "8ef20d61-4e5c-48e8-947c-2429fcbfe059" (UID: "8ef20d61-4e5c-48e8-947c-2429fcbfe059"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:29:10.005850 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:10.005818 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ef20d61-4e5c-48e8-947c-2429fcbfe059-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:29:10.005850 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:10.005853 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cwv7\" (UniqueName: \"kubernetes.io/projected/8ef20d61-4e5c-48e8-947c-2429fcbfe059-kube-api-access-2cwv7\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:29:10.006097 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:10.005869 4861 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ef20d61-4e5c-48e8-947c-2429fcbfe059-webhook-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:29:10.470663 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:10.470632 4861 generic.go:356] "Generic (PLEG): container finished" podID="8ef20d61-4e5c-48e8-947c-2429fcbfe059" containerID="3a2dbb28daf785f1c19679088752948f10485d1dc069adfabcc1dc5ef7630231" exitCode=0 Oct 30 16:29:10.471180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:10.470684 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5656d77c94-s4jnw" Oct 30 16:29:10.471180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:10.470712 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5656d77c94-s4jnw" event={"ID":"8ef20d61-4e5c-48e8-947c-2429fcbfe059","Type":"ContainerDied","Data":"3a2dbb28daf785f1c19679088752948f10485d1dc069adfabcc1dc5ef7630231"} Oct 30 16:29:10.471180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:10.470785 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5656d77c94-s4jnw" event={"ID":"8ef20d61-4e5c-48e8-947c-2429fcbfe059","Type":"ContainerDied","Data":"83d60b587b04ebdca5872dd78f79e35f4d20f45d6c69402406d417a904588b6e"} Oct 30 16:29:10.471180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:10.470808 4861 scope.go:122] "RemoveContainer" containerID="3a2dbb28daf785f1c19679088752948f10485d1dc069adfabcc1dc5ef7630231" Oct 30 16:29:10.480266 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:10.480249 4861 scope.go:122] "RemoveContainer" containerID="3a2dbb28daf785f1c19679088752948f10485d1dc069adfabcc1dc5ef7630231" Oct 30 16:29:10.480541 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:29:10.480524 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a2dbb28daf785f1c19679088752948f10485d1dc069adfabcc1dc5ef7630231\": container with ID starting with 3a2dbb28daf785f1c19679088752948f10485d1dc069adfabcc1dc5ef7630231 not found: ID does not exist" containerID="3a2dbb28daf785f1c19679088752948f10485d1dc069adfabcc1dc5ef7630231" Oct 30 16:29:10.480620 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:10.480548 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a2dbb28daf785f1c19679088752948f10485d1dc069adfabcc1dc5ef7630231"} err="failed to get container status \"3a2dbb28daf785f1c19679088752948f10485d1dc069adfabcc1dc5ef7630231\": rpc error: code = NotFound desc = could not find container \"3a2dbb28daf785f1c19679088752948f10485d1dc069adfabcc1dc5ef7630231\": container with ID starting with 3a2dbb28daf785f1c19679088752948f10485d1dc069adfabcc1dc5ef7630231 not found: ID does not exist" Oct 30 16:29:10.523404 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:10.523376 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5656d77c94-s4jnw"] Oct 30 16:29:10.536193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:10.536170 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5656d77c94-s4jnw"] Oct 30 16:29:11.805437 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:11.805407 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ef20d61-4e5c-48e8-947c-2429fcbfe059" path="/var/lib/kubelet/pods/8ef20d61-4e5c-48e8-947c-2429fcbfe059/volumes" Oct 30 16:29:12.482981 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:12.482947 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" event={"ID":"be3b019c-c736-4857-bda5-92df56b650c5","Type":"ContainerStarted","Data":"48dc4003caea6f14ee24e5d854c3804c5dbc80175e262214c29db2ddbbdb7933"} Oct 30 16:29:12.483167 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:12.483088 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" Oct 30 16:29:12.527404 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:12.527349 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" podStartSLOduration=1.504300392 podStartE2EDuration="12.527335563s" totalImagesPullingTime="11.023035171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:29:12.526798535 +0000 UTC m=+382.869472564" watchObservedRunningTime="2026-10-30 16:29:12.527335563 +0000 UTC m=+382.870009601" Oct 30 16:29:20.290832 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:29:20.290787 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:29:23.492573 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:23.492537 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-klj74" Oct 30 16:29:26.656439 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:26.656411 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:27.781707 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:27.781670 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:28.115808 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.115709 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:28.252282 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.252247 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:28.416337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.416266 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:28.565476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.565444 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d"] Oct 30 16:29:28.565839 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.565824 4861 cpu_manager.go:414] "containerMap: removing container" podUID="8ef20d61-4e5c-48e8-947c-2429fcbfe059" containerName="manager" Oct 30 16:29:28.565839 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.565839 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef20d61-4e5c-48e8-947c-2429fcbfe059" containerName="manager" Oct 30 16:29:28.565938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.565912 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef20d61-4e5c-48e8-947c-2429fcbfe059" containerName="manager" Oct 30 16:29:28.694118 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.694042 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d"] Oct 30 16:29:28.694259 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.694148 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:28.721521 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.721499 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 16:29:28.721798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.721752 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 16:29:28.721900 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.721888 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-k46zs\"" Oct 30 16:29:28.855871 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.855841 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjsbw\" (UniqueName: \"kubernetes.io/projected/facfd2bf-5e2d-49b2-8583-8445c7104987-kube-api-access-tjsbw\") pod \"maas-api-57cfb7d588-qv85d\" (UID: \"facfd2bf-5e2d-49b2-8583-8445c7104987\") " pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:28.855871 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.855874 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/facfd2bf-5e2d-49b2-8583-8445c7104987-metrics-certs\") pod \"maas-api-57cfb7d588-qv85d\" (UID: \"facfd2bf-5e2d-49b2-8583-8445c7104987\") " pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:28.856264 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.855990 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/facfd2bf-5e2d-49b2-8583-8445c7104987-maas-api-tls\") pod \"maas-api-57cfb7d588-qv85d\" (UID: \"facfd2bf-5e2d-49b2-8583-8445c7104987\") " pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:28.856264 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.856028 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/facfd2bf-5e2d-49b2-8583-8445c7104987-openshift-service-ca\") pod \"maas-api-57cfb7d588-qv85d\" (UID: \"facfd2bf-5e2d-49b2-8583-8445c7104987\") " pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:28.957331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.957262 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/facfd2bf-5e2d-49b2-8583-8445c7104987-metrics-certs\") pod \"maas-api-57cfb7d588-qv85d\" (UID: \"facfd2bf-5e2d-49b2-8583-8445c7104987\") " pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:28.957464 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.957358 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/facfd2bf-5e2d-49b2-8583-8445c7104987-maas-api-tls\") pod \"maas-api-57cfb7d588-qv85d\" (UID: \"facfd2bf-5e2d-49b2-8583-8445c7104987\") " pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:28.957464 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.957390 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/facfd2bf-5e2d-49b2-8583-8445c7104987-openshift-service-ca\") pod \"maas-api-57cfb7d588-qv85d\" (UID: \"facfd2bf-5e2d-49b2-8583-8445c7104987\") " pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:28.957464 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.957428 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjsbw\" (UniqueName: \"kubernetes.io/projected/facfd2bf-5e2d-49b2-8583-8445c7104987-kube-api-access-tjsbw\") pod \"maas-api-57cfb7d588-qv85d\" (UID: \"facfd2bf-5e2d-49b2-8583-8445c7104987\") " pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:28.958386 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.958287 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/facfd2bf-5e2d-49b2-8583-8445c7104987-openshift-service-ca\") pod \"maas-api-57cfb7d588-qv85d\" (UID: \"facfd2bf-5e2d-49b2-8583-8445c7104987\") " pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:28.960121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.960097 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/facfd2bf-5e2d-49b2-8583-8445c7104987-metrics-certs\") pod \"maas-api-57cfb7d588-qv85d\" (UID: \"facfd2bf-5e2d-49b2-8583-8445c7104987\") " pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:28.960239 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.960152 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/facfd2bf-5e2d-49b2-8583-8445c7104987-maas-api-tls\") pod \"maas-api-57cfb7d588-qv85d\" (UID: \"facfd2bf-5e2d-49b2-8583-8445c7104987\") " pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:28.980807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:28.980788 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjsbw\" (UniqueName: \"kubernetes.io/projected/facfd2bf-5e2d-49b2-8583-8445c7104987-kube-api-access-tjsbw\") pod \"maas-api-57cfb7d588-qv85d\" (UID: \"facfd2bf-5e2d-49b2-8583-8445c7104987\") " pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:29.039717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:29.039697 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:29.390530 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:29.390500 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d"] Oct 30 16:29:29.393936 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:29:29.393908 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacfd2bf_5e2d_49b2_8583_8445c7104987.slice/crio-1632bfa0b5a85a60daa47a085b6dde8f069fb880adee1b2973c7a18e1a61dcfe WatchSource:0}: Error finding container 1632bfa0b5a85a60daa47a085b6dde8f069fb880adee1b2973c7a18e1a61dcfe: Status 404 returned error can't find the container with id 1632bfa0b5a85a60daa47a085b6dde8f069fb880adee1b2973c7a18e1a61dcfe Oct 30 16:29:29.553369 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:29.553332 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" event={"ID":"facfd2bf-5e2d-49b2-8583-8445c7104987","Type":"ContainerStarted","Data":"1632bfa0b5a85a60daa47a085b6dde8f069fb880adee1b2973c7a18e1a61dcfe"} Oct 30 16:29:31.564788 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:31.564750 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" event={"ID":"facfd2bf-5e2d-49b2-8583-8445c7104987","Type":"ContainerStarted","Data":"3fd48d75bf501be8e5fdb87128b55d65ff6b44f346caa0a0eb348a6308adaf5e"} Oct 30 16:29:31.565178 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:31.564891 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:31.608761 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:31.608653 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" podStartSLOduration=1.650092628 podStartE2EDuration="3.608639783s" totalImagesPullingTime="1.958547155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:29:31.60814068 +0000 UTC m=+401.950814720" watchObservedRunningTime="2026-10-30 16:29:31.608639783 +0000 UTC m=+401.951313821" Oct 30 16:29:37.574945 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:37.574910 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:38.064173 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.064138 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:38.200558 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.200508 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:38.552288 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.552258 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv"] Oct 30 16:29:38.555875 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.555855 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" Oct 30 16:29:38.588990 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.588964 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv"] Oct 30 16:29:38.634470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.634434 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f59b77e-b7c6-4e20-b138-cee7899675fc-maas-api-tls\") pod \"maas-api-6f99ff7476-mb6nv\" (UID: \"5f59b77e-b7c6-4e20-b138-cee7899675fc\") " pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" Oct 30 16:29:38.634590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.634506 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8x9w\" (UniqueName: \"kubernetes.io/projected/5f59b77e-b7c6-4e20-b138-cee7899675fc-kube-api-access-m8x9w\") pod \"maas-api-6f99ff7476-mb6nv\" (UID: \"5f59b77e-b7c6-4e20-b138-cee7899675fc\") " pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" Oct 30 16:29:38.634634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.634593 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f59b77e-b7c6-4e20-b138-cee7899675fc-metrics-certs\") pod \"maas-api-6f99ff7476-mb6nv\" (UID: \"5f59b77e-b7c6-4e20-b138-cee7899675fc\") " pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" Oct 30 16:29:38.634634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.634621 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f59b77e-b7c6-4e20-b138-cee7899675fc-openshift-service-ca\") pod \"maas-api-6f99ff7476-mb6nv\" (UID: \"5f59b77e-b7c6-4e20-b138-cee7899675fc\") " pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" Oct 30 16:29:38.735845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.735815 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f59b77e-b7c6-4e20-b138-cee7899675fc-maas-api-tls\") pod \"maas-api-6f99ff7476-mb6nv\" (UID: \"5f59b77e-b7c6-4e20-b138-cee7899675fc\") " pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" Oct 30 16:29:38.735969 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.735868 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8x9w\" (UniqueName: \"kubernetes.io/projected/5f59b77e-b7c6-4e20-b138-cee7899675fc-kube-api-access-m8x9w\") pod \"maas-api-6f99ff7476-mb6nv\" (UID: \"5f59b77e-b7c6-4e20-b138-cee7899675fc\") " pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" Oct 30 16:29:38.736039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.735992 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f59b77e-b7c6-4e20-b138-cee7899675fc-metrics-certs\") pod \"maas-api-6f99ff7476-mb6nv\" (UID: \"5f59b77e-b7c6-4e20-b138-cee7899675fc\") " pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" Oct 30 16:29:38.736090 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.736034 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f59b77e-b7c6-4e20-b138-cee7899675fc-openshift-service-ca\") pod \"maas-api-6f99ff7476-mb6nv\" (UID: \"5f59b77e-b7c6-4e20-b138-cee7899675fc\") " pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" Oct 30 16:29:38.736685 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.736664 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f59b77e-b7c6-4e20-b138-cee7899675fc-openshift-service-ca\") pod \"maas-api-6f99ff7476-mb6nv\" (UID: \"5f59b77e-b7c6-4e20-b138-cee7899675fc\") " pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" Oct 30 16:29:38.738647 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.738629 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f59b77e-b7c6-4e20-b138-cee7899675fc-metrics-certs\") pod \"maas-api-6f99ff7476-mb6nv\" (UID: \"5f59b77e-b7c6-4e20-b138-cee7899675fc\") " pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" Oct 30 16:29:38.738703 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.738681 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f59b77e-b7c6-4e20-b138-cee7899675fc-maas-api-tls\") pod \"maas-api-6f99ff7476-mb6nv\" (UID: \"5f59b77e-b7c6-4e20-b138-cee7899675fc\") " pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" Oct 30 16:29:38.770160 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.770130 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8x9w\" (UniqueName: \"kubernetes.io/projected/5f59b77e-b7c6-4e20-b138-cee7899675fc-kube-api-access-m8x9w\") pod \"maas-api-6f99ff7476-mb6nv\" (UID: \"5f59b77e-b7c6-4e20-b138-cee7899675fc\") " pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" Oct 30 16:29:38.867627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:38.867571 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" Oct 30 16:29:39.166374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:39.166309 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv"] Oct 30 16:29:39.169453 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:29:39.169427 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f59b77e_b7c6_4e20_b138_cee7899675fc.slice/crio-ddc71471c7966632e501cd0e075f3e0f57923311bfb8c22bde81918ebcc072e5 WatchSource:0}: Error finding container ddc71471c7966632e501cd0e075f3e0f57923311bfb8c22bde81918ebcc072e5: Status 404 returned error can't find the container with id ddc71471c7966632e501cd0e075f3e0f57923311bfb8c22bde81918ebcc072e5 Oct 30 16:29:39.602219 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:39.602176 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" event={"ID":"5f59b77e-b7c6-4e20-b138-cee7899675fc","Type":"ContainerStarted","Data":"ddc71471c7966632e501cd0e075f3e0f57923311bfb8c22bde81918ebcc072e5"} Oct 30 16:29:40.539379 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:40.539340 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:29:40.608346 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:40.608314 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" event={"ID":"5f59b77e-b7c6-4e20-b138-cee7899675fc","Type":"ContainerStarted","Data":"631b0f587dc25a61395b4ede245df7c77425c8e30b6a822c873f7eb5eede3037"} Oct 30 16:29:40.608779 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:40.608425 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" Oct 30 16:29:40.652039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:40.651995 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" podStartSLOduration=2.262824475 podStartE2EDuration="2.651984895s" totalImagesPullingTime="389.16042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:29:40.651439085 +0000 UTC m=+410.994113123" watchObservedRunningTime="2026-10-30 16:29:40.651984895 +0000 UTC m=+410.994658934" Oct 30 16:29:46.619262 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:46.619230 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6f99ff7476-mb6nv" Oct 30 16:29:46.695460 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:46.695423 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d"] Oct 30 16:29:46.695672 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:46.695646 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" podUID="facfd2bf-5e2d-49b2-8583-8445c7104987" containerName="maas-api" containerID="cri-o://3fd48d75bf501be8e5fdb87128b55d65ff6b44f346caa0a0eb348a6308adaf5e" gracePeriod=30 Oct 30 16:29:46.942435 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:46.942414 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:47.107437 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.107406 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/facfd2bf-5e2d-49b2-8583-8445c7104987-kube-api-access-tjsbw\" (UniqueName: \"kubernetes.io/projected/facfd2bf-5e2d-49b2-8583-8445c7104987-kube-api-access-tjsbw\") pod \"facfd2bf-5e2d-49b2-8583-8445c7104987\" (UID: \"facfd2bf-5e2d-49b2-8583-8445c7104987\") " Oct 30 16:29:47.107622 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.107445 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/facfd2bf-5e2d-49b2-8583-8445c7104987-metrics-certs\" (UniqueName: \"kubernetes.io/secret/facfd2bf-5e2d-49b2-8583-8445c7104987-metrics-certs\") pod \"facfd2bf-5e2d-49b2-8583-8445c7104987\" (UID: \"facfd2bf-5e2d-49b2-8583-8445c7104987\") " Oct 30 16:29:47.107622 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.107468 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/facfd2bf-5e2d-49b2-8583-8445c7104987-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/facfd2bf-5e2d-49b2-8583-8445c7104987-maas-api-tls\") pod \"facfd2bf-5e2d-49b2-8583-8445c7104987\" (UID: \"facfd2bf-5e2d-49b2-8583-8445c7104987\") " Oct 30 16:29:47.107622 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.107503 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/facfd2bf-5e2d-49b2-8583-8445c7104987-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/facfd2bf-5e2d-49b2-8583-8445c7104987-openshift-service-ca\") pod \"facfd2bf-5e2d-49b2-8583-8445c7104987\" (UID: \"facfd2bf-5e2d-49b2-8583-8445c7104987\") " Oct 30 16:29:47.107959 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.107935 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/facfd2bf-5e2d-49b2-8583-8445c7104987-openshift-service-ca" pod "facfd2bf-5e2d-49b2-8583-8445c7104987" (UID: "facfd2bf-5e2d-49b2-8583-8445c7104987"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:29:47.109797 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.109760 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/facfd2bf-5e2d-49b2-8583-8445c7104987-metrics-certs" pod "facfd2bf-5e2d-49b2-8583-8445c7104987" (UID: "facfd2bf-5e2d-49b2-8583-8445c7104987"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:29:47.109897 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.109802 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/facfd2bf-5e2d-49b2-8583-8445c7104987-maas-api-tls" pod "facfd2bf-5e2d-49b2-8583-8445c7104987" (UID: "facfd2bf-5e2d-49b2-8583-8445c7104987"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:29:47.109897 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.109875 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/facfd2bf-5e2d-49b2-8583-8445c7104987-kube-api-access-tjsbw" pod "facfd2bf-5e2d-49b2-8583-8445c7104987" (UID: "facfd2bf-5e2d-49b2-8583-8445c7104987"). InnerVolumeSpecName "kube-api-access-tjsbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:29:47.208770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.208697 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjsbw\" (UniqueName: \"kubernetes.io/projected/facfd2bf-5e2d-49b2-8583-8445c7104987-kube-api-access-tjsbw\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:29:47.208770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.208749 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/facfd2bf-5e2d-49b2-8583-8445c7104987-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:29:47.208770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.208760 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/facfd2bf-5e2d-49b2-8583-8445c7104987-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:29:47.208770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.208768 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/facfd2bf-5e2d-49b2-8583-8445c7104987-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:29:47.643604 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.643567 4861 generic.go:356] "Generic (PLEG): container finished" podID="facfd2bf-5e2d-49b2-8583-8445c7104987" containerID="3fd48d75bf501be8e5fdb87128b55d65ff6b44f346caa0a0eb348a6308adaf5e" exitCode=0 Oct 30 16:29:47.644053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.643622 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" Oct 30 16:29:47.644053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.643657 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" event={"ID":"facfd2bf-5e2d-49b2-8583-8445c7104987","Type":"ContainerDied","Data":"3fd48d75bf501be8e5fdb87128b55d65ff6b44f346caa0a0eb348a6308adaf5e"} Oct 30 16:29:47.644053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.643738 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d" event={"ID":"facfd2bf-5e2d-49b2-8583-8445c7104987","Type":"ContainerDied","Data":"1632bfa0b5a85a60daa47a085b6dde8f069fb880adee1b2973c7a18e1a61dcfe"} Oct 30 16:29:47.644053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.643773 4861 scope.go:122] "RemoveContainer" containerID="3fd48d75bf501be8e5fdb87128b55d65ff6b44f346caa0a0eb348a6308adaf5e" Oct 30 16:29:47.653545 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.653529 4861 scope.go:122] "RemoveContainer" containerID="3fd48d75bf501be8e5fdb87128b55d65ff6b44f346caa0a0eb348a6308adaf5e" Oct 30 16:29:47.653886 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:29:47.653869 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd48d75bf501be8e5fdb87128b55d65ff6b44f346caa0a0eb348a6308adaf5e\": container with ID starting with 3fd48d75bf501be8e5fdb87128b55d65ff6b44f346caa0a0eb348a6308adaf5e not found: ID does not exist" containerID="3fd48d75bf501be8e5fdb87128b55d65ff6b44f346caa0a0eb348a6308adaf5e" Oct 30 16:29:47.653962 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.653897 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd48d75bf501be8e5fdb87128b55d65ff6b44f346caa0a0eb348a6308adaf5e"} err="failed to get container status \"3fd48d75bf501be8e5fdb87128b55d65ff6b44f346caa0a0eb348a6308adaf5e\": rpc error: code = NotFound desc = could not find container \"3fd48d75bf501be8e5fdb87128b55d65ff6b44f346caa0a0eb348a6308adaf5e\": container with ID starting with 3fd48d75bf501be8e5fdb87128b55d65ff6b44f346caa0a0eb348a6308adaf5e not found: ID does not exist" Oct 30 16:29:47.695322 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.695299 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d"] Oct 30 16:29:47.707557 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.707500 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-57cfb7d588-qv85d"] Oct 30 16:29:47.804828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:29:47.804801 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="facfd2bf-5e2d-49b2-8583-8445c7104987" path="/var/lib/kubelet/pods/facfd2bf-5e2d-49b2-8583-8445c7104987/volumes" Oct 30 16:29:50.288847 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:29:50.288803 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:30:20.288524 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:30:20.288488 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:30:35.368603 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.368561 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2w4zs"] Oct 30 16:30:35.369068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.368920 4861 cpu_manager.go:414] "containerMap: removing container" podUID="facfd2bf-5e2d-49b2-8583-8445c7104987" containerName="maas-api" Oct 30 16:30:35.369068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.368931 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="facfd2bf-5e2d-49b2-8583-8445c7104987" containerName="maas-api" Oct 30 16:30:35.369068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.369008 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="facfd2bf-5e2d-49b2-8583-8445c7104987" containerName="maas-api" Oct 30 16:30:35.372121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.372105 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" Oct 30 16:30:35.385163 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.385118 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:35.389445 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.389423 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 16:30:35.390837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.390817 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2w4zs"] Oct 30 16:30:35.401838 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.401814 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-66cwb\"" Oct 30 16:30:35.401838 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.401839 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 16:30:35.480777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.480739 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663eb5d7-3331-4b7b-874f-1eba2aab60ef-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-2w4zs\" (UID: \"663eb5d7-3331-4b7b-874f-1eba2aab60ef\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" Oct 30 16:30:35.480932 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.480801 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n57w\" (UniqueName: \"kubernetes.io/projected/663eb5d7-3331-4b7b-874f-1eba2aab60ef-kube-api-access-7n57w\") pod \"ai-gateway-controller-7d85675f98-2w4zs\" (UID: \"663eb5d7-3331-4b7b-874f-1eba2aab60ef\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" Oct 30 16:30:35.533030 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.532980 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:35.582117 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.582087 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663eb5d7-3331-4b7b-874f-1eba2aab60ef-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-2w4zs\" (UID: \"663eb5d7-3331-4b7b-874f-1eba2aab60ef\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" Oct 30 16:30:35.582306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.582129 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n57w\" (UniqueName: \"kubernetes.io/projected/663eb5d7-3331-4b7b-874f-1eba2aab60ef-kube-api-access-7n57w\") pod \"ai-gateway-controller-7d85675f98-2w4zs\" (UID: \"663eb5d7-3331-4b7b-874f-1eba2aab60ef\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" Oct 30 16:30:35.584826 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.584803 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663eb5d7-3331-4b7b-874f-1eba2aab60ef-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-2w4zs\" (UID: \"663eb5d7-3331-4b7b-874f-1eba2aab60ef\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" Oct 30 16:30:35.605917 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.605895 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n57w\" (UniqueName: \"kubernetes.io/projected/663eb5d7-3331-4b7b-874f-1eba2aab60ef-kube-api-access-7n57w\") pod \"ai-gateway-controller-7d85675f98-2w4zs\" (UID: \"663eb5d7-3331-4b7b-874f-1eba2aab60ef\") " pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" Oct 30 16:30:35.684953 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.684892 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" Oct 30 16:30:35.986654 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:35.986573 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2w4zs"] Oct 30 16:30:35.989351 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:30:35.989316 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663eb5d7_3331_4b7b_874f_1eba2aab60ef.slice/crio-9af8ffec70ef2656e1ae9eb606acc9c60a6122da6a9662a2edb2bc157d73436e WatchSource:0}: Error finding container 9af8ffec70ef2656e1ae9eb606acc9c60a6122da6a9662a2edb2bc157d73436e: Status 404 returned error can't find the container with id 9af8ffec70ef2656e1ae9eb606acc9c60a6122da6a9662a2edb2bc157d73436e Oct 30 16:30:36.861223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:36.861186 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" event={"ID":"663eb5d7-3331-4b7b-874f-1eba2aab60ef","Type":"ContainerStarted","Data":"9af8ffec70ef2656e1ae9eb606acc9c60a6122da6a9662a2edb2bc157d73436e"} Oct 30 16:30:37.866500 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:37.866421 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" event={"ID":"663eb5d7-3331-4b7b-874f-1eba2aab60ef","Type":"ContainerStarted","Data":"2642314857cbfa2c34f8b9cbf44dbf6ec1e07cabf9ab3d3e77ff2c71ebf94c64"} Oct 30 16:30:37.866868 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:37.866499 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" Oct 30 16:30:37.908901 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:37.908851 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" podStartSLOduration=1.386817203 podStartE2EDuration="2.908838026s" totalImagesPullingTime="1.522020823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:30:37.907849495 +0000 UTC m=+468.250523532" watchObservedRunningTime="2026-10-30 16:30:37.908838026 +0000 UTC m=+468.251512065" Oct 30 16:30:38.716243 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:38.716203 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:38.794594 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:38.794563 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-gp6rg"] Oct 30 16:30:38.798147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:38.798131 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-gp6rg" Oct 30 16:30:38.817956 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:38.817933 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 16:30:38.832880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:38.832850 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-gp6rg"] Oct 30 16:30:38.858866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:38.858845 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zpxq6"] Oct 30 16:30:38.862186 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:38.862171 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zpxq6" Oct 30 16:30:38.868658 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:38.868612 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:38.881872 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:38.881852 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-x8gjp\"" Oct 30 16:30:38.884074 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:38.884056 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zpxq6"] Oct 30 16:30:38.911805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:38.911783 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6nts\" (UniqueName: \"kubernetes.io/projected/a2319b60-ec05-4f37-b97a-79de3c482d58-kube-api-access-z6nts\") pod \"payload-processing-5f78687d5b-gp6rg\" (UID: \"a2319b60-ec05-4f37-b97a-79de3c482d58\") " pod="openshift-ingress/payload-processing-5f78687d5b-gp6rg" Oct 30 16:30:38.911941 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:38.911841 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2319b60-ec05-4f37-b97a-79de3c482d58-plugins-config-volume\") pod \"payload-processing-5f78687d5b-gp6rg\" (UID: \"a2319b60-ec05-4f37-b97a-79de3c482d58\") " pod="openshift-ingress/payload-processing-5f78687d5b-gp6rg" Oct 30 16:30:39.013083 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:39.013054 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2319b60-ec05-4f37-b97a-79de3c482d58-plugins-config-volume\") pod \"payload-processing-5f78687d5b-gp6rg\" (UID: \"a2319b60-ec05-4f37-b97a-79de3c482d58\") " pod="openshift-ingress/payload-processing-5f78687d5b-gp6rg" Oct 30 16:30:39.013217 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:39.013163 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6nts\" (UniqueName: \"kubernetes.io/projected/a2319b60-ec05-4f37-b97a-79de3c482d58-kube-api-access-z6nts\") pod \"payload-processing-5f78687d5b-gp6rg\" (UID: \"a2319b60-ec05-4f37-b97a-79de3c482d58\") " pod="openshift-ingress/payload-processing-5f78687d5b-gp6rg" Oct 30 16:30:39.013333 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:39.013303 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57f57e8-e81b-476f-8934-53a8cc63b937-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zpxq6\" (UID: \"a57f57e8-e81b-476f-8934-53a8cc63b937\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zpxq6" Oct 30 16:30:39.013441 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:39.013418 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwz59\" (UniqueName: \"kubernetes.io/projected/a57f57e8-e81b-476f-8934-53a8cc63b937-kube-api-access-nwz59\") pod \"payload-pre-processing-7b98bb678c-zpxq6\" (UID: \"a57f57e8-e81b-476f-8934-53a8cc63b937\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zpxq6" Oct 30 16:30:39.013834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:39.013817 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2319b60-ec05-4f37-b97a-79de3c482d58-plugins-config-volume\") pod \"payload-processing-5f78687d5b-gp6rg\" (UID: \"a2319b60-ec05-4f37-b97a-79de3c482d58\") " pod="openshift-ingress/payload-processing-5f78687d5b-gp6rg" Oct 30 16:30:39.036888 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:39.036863 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6nts\" (UniqueName: \"kubernetes.io/projected/a2319b60-ec05-4f37-b97a-79de3c482d58-kube-api-access-z6nts\") pod \"payload-processing-5f78687d5b-gp6rg\" (UID: \"a2319b60-ec05-4f37-b97a-79de3c482d58\") " pod="openshift-ingress/payload-processing-5f78687d5b-gp6rg" Oct 30 16:30:39.110699 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:39.110666 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-gp6rg" Oct 30 16:30:39.114188 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:39.113902 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57f57e8-e81b-476f-8934-53a8cc63b937-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zpxq6\" (UID: \"a57f57e8-e81b-476f-8934-53a8cc63b937\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zpxq6" Oct 30 16:30:39.114188 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:39.113965 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwz59\" (UniqueName: \"kubernetes.io/projected/a57f57e8-e81b-476f-8934-53a8cc63b937-kube-api-access-nwz59\") pod \"payload-pre-processing-7b98bb678c-zpxq6\" (UID: \"a57f57e8-e81b-476f-8934-53a8cc63b937\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zpxq6" Oct 30 16:30:39.114790 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:39.114768 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a57f57e8-e81b-476f-8934-53a8cc63b937-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-zpxq6\" (UID: \"a57f57e8-e81b-476f-8934-53a8cc63b937\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zpxq6" Oct 30 16:30:39.137818 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:39.137789 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwz59\" (UniqueName: \"kubernetes.io/projected/a57f57e8-e81b-476f-8934-53a8cc63b937-kube-api-access-nwz59\") pod \"payload-pre-processing-7b98bb678c-zpxq6\" (UID: \"a57f57e8-e81b-476f-8934-53a8cc63b937\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-zpxq6" Oct 30 16:30:39.175125 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:39.175094 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zpxq6" Oct 30 16:30:39.412124 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:39.412059 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-gp6rg"] Oct 30 16:30:39.414690 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:30:39.414668 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2319b60_ec05_4f37_b97a_79de3c482d58.slice/crio-f47379da60b2a128b1bb3199cebe8b81f1432c27476d0d35aa0002eeb9f4a7ff WatchSource:0}: Error finding container f47379da60b2a128b1bb3199cebe8b81f1432c27476d0d35aa0002eeb9f4a7ff: Status 404 returned error can't find the container with id f47379da60b2a128b1bb3199cebe8b81f1432c27476d0d35aa0002eeb9f4a7ff Oct 30 16:30:39.483945 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:39.483919 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-zpxq6"] Oct 30 16:30:39.485649 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:30:39.485625 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda57f57e8_e81b_476f_8934_53a8cc63b937.slice/crio-f8b6fac7caf0d86538fcb2781b13fb6b663fedafbb5ecf498ba73bca506182ab WatchSource:0}: Error finding container f8b6fac7caf0d86538fcb2781b13fb6b663fedafbb5ecf498ba73bca506182ab: Status 404 returned error can't find the container with id f8b6fac7caf0d86538fcb2781b13fb6b663fedafbb5ecf498ba73bca506182ab Oct 30 16:30:39.878271 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:39.878236 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-gp6rg" event={"ID":"a2319b60-ec05-4f37-b97a-79de3c482d58","Type":"ContainerStarted","Data":"f47379da60b2a128b1bb3199cebe8b81f1432c27476d0d35aa0002eeb9f4a7ff"} Oct 30 16:30:39.879625 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:39.879601 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zpxq6" event={"ID":"a57f57e8-e81b-476f-8934-53a8cc63b937","Type":"ContainerStarted","Data":"f8b6fac7caf0d86538fcb2781b13fb6b663fedafbb5ecf498ba73bca506182ab"} Oct 30 16:30:40.671974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:40.671924 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:30:41.890878 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:41.890834 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-gp6rg" event={"ID":"a2319b60-ec05-4f37-b97a-79de3c482d58","Type":"ContainerStarted","Data":"4227d7d1b58d367790e411633183310232dbce8458cd674cf420b7f8dc398477"} Oct 30 16:30:41.891295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:41.890897 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-gp6rg" Oct 30 16:30:41.892507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:41.892478 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zpxq6" event={"ID":"a57f57e8-e81b-476f-8934-53a8cc63b937","Type":"ContainerStarted","Data":"b9cd915bb4ab4df4ef28fd4c05d460b4aed183ca134e2e9fdb261360c12f0624"} Oct 30 16:30:41.892640 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:41.892533 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zpxq6" Oct 30 16:30:41.935119 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:41.935074 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-gp6rg" podStartSLOduration=1.649500221 podStartE2EDuration="3.935061947s" totalImagesPullingTime="2.285561726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:30:41.933439018 +0000 UTC m=+472.276113048" watchObservedRunningTime="2026-10-30 16:30:41.935061947 +0000 UTC m=+472.277736194" Oct 30 16:30:41.974556 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:41.974512 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zpxq6" podStartSLOduration=1.727877522 podStartE2EDuration="3.974498973s" totalImagesPullingTime="2.246621451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:30:41.973573837 +0000 UTC m=+472.316247874" watchObservedRunningTime="2026-10-30 16:30:41.974498973 +0000 UTC m=+472.317173010" Oct 30 16:30:48.874441 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:48.874412 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" Oct 30 16:30:49.154263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:49.154194 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7f4d877496-xgjn8"] Oct 30 16:30:49.157847 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:49.157829 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7f4d877496-xgjn8" Oct 30 16:30:49.179453 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:49.179430 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7f4d877496-xgjn8"] Oct 30 16:30:49.292926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:49.292887 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgm9r\" (UniqueName: \"kubernetes.io/projected/d8b4d873-2506-45c2-ad31-ed4b0b503ee8-kube-api-access-tgm9r\") pod \"ai-gateway-controller-7f4d877496-xgjn8\" (UID: \"d8b4d873-2506-45c2-ad31-ed4b0b503ee8\") " pod="opendatahub/ai-gateway-controller-7f4d877496-xgjn8" Oct 30 16:30:49.293093 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:49.292948 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8b4d873-2506-45c2-ad31-ed4b0b503ee8-metrics-certs\") pod \"ai-gateway-controller-7f4d877496-xgjn8\" (UID: \"d8b4d873-2506-45c2-ad31-ed4b0b503ee8\") " pod="opendatahub/ai-gateway-controller-7f4d877496-xgjn8" Oct 30 16:30:49.393624 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:49.393597 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8b4d873-2506-45c2-ad31-ed4b0b503ee8-metrics-certs\") pod \"ai-gateway-controller-7f4d877496-xgjn8\" (UID: \"d8b4d873-2506-45c2-ad31-ed4b0b503ee8\") " pod="opendatahub/ai-gateway-controller-7f4d877496-xgjn8" Oct 30 16:30:49.393752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:49.393658 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgm9r\" (UniqueName: \"kubernetes.io/projected/d8b4d873-2506-45c2-ad31-ed4b0b503ee8-kube-api-access-tgm9r\") pod \"ai-gateway-controller-7f4d877496-xgjn8\" (UID: \"d8b4d873-2506-45c2-ad31-ed4b0b503ee8\") " pod="opendatahub/ai-gateway-controller-7f4d877496-xgjn8" Oct 30 16:30:49.396255 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:49.396237 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8b4d873-2506-45c2-ad31-ed4b0b503ee8-metrics-certs\") pod \"ai-gateway-controller-7f4d877496-xgjn8\" (UID: \"d8b4d873-2506-45c2-ad31-ed4b0b503ee8\") " pod="opendatahub/ai-gateway-controller-7f4d877496-xgjn8" Oct 30 16:30:49.417473 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:49.417419 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgm9r\" (UniqueName: \"kubernetes.io/projected/d8b4d873-2506-45c2-ad31-ed4b0b503ee8-kube-api-access-tgm9r\") pod \"ai-gateway-controller-7f4d877496-xgjn8\" (UID: \"d8b4d873-2506-45c2-ad31-ed4b0b503ee8\") " pod="opendatahub/ai-gateway-controller-7f4d877496-xgjn8" Oct 30 16:30:49.470022 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:49.469993 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7f4d877496-xgjn8" Oct 30 16:30:49.771353 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:49.771321 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7f4d877496-xgjn8"] Oct 30 16:30:49.774994 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:30:49.774968 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b4d873_2506_45c2_ad31_ed4b0b503ee8.slice/crio-9edff51eb2aa9781c64ff7dccc10439d9b5970f8b960d6f8c087424cff025964 WatchSource:0}: Error finding container 9edff51eb2aa9781c64ff7dccc10439d9b5970f8b960d6f8c087424cff025964: Status 404 returned error can't find the container with id 9edff51eb2aa9781c64ff7dccc10439d9b5970f8b960d6f8c087424cff025964 Oct 30 16:30:49.932347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:49.932317 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7f4d877496-xgjn8" event={"ID":"d8b4d873-2506-45c2-ad31-ed4b0b503ee8","Type":"ContainerStarted","Data":"9edff51eb2aa9781c64ff7dccc10439d9b5970f8b960d6f8c087424cff025964"} Oct 30 16:30:50.291225 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:30:50.291186 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:30:50.938965 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:50.938917 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7f4d877496-xgjn8" event={"ID":"d8b4d873-2506-45c2-ad31-ed4b0b503ee8","Type":"ContainerStarted","Data":"89aaaee9be2a87e8a4e85d295148297e7becdcd48acfabf55bc0a05299a90cbe"} Oct 30 16:30:50.939407 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:50.939000 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7f4d877496-xgjn8" Oct 30 16:30:50.982043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:50.981995 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7f4d877496-xgjn8" podStartSLOduration=1.6322465670000001 podStartE2EDuration="1.981982728s" totalImagesPullingTime="349.736161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:30:50.98093992 +0000 UTC m=+481.323613959" watchObservedRunningTime="2026-10-30 16:30:50.981982728 +0000 UTC m=+481.324656766" Oct 30 16:30:52.900640 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:52.900611 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-gp6rg" Oct 30 16:30:52.901155 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:30:52.900768 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-zpxq6" Oct 30 16:31:01.946791 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:01.946681 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7f4d877496-xgjn8" Oct 30 16:31:02.021624 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:02.021594 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2w4zs"] Oct 30 16:31:02.021844 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:02.021803 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" podUID="663eb5d7-3331-4b7b-874f-1eba2aab60ef" containerName="manager" containerID="cri-o://2642314857cbfa2c34f8b9cbf44dbf6ec1e07cabf9ab3d3e77ff2c71ebf94c64" gracePeriod=30 Oct 30 16:31:02.270042 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:02.270018 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" Oct 30 16:31:02.395057 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:02.395026 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/663eb5d7-3331-4b7b-874f-1eba2aab60ef-kube-api-access-7n57w\" (UniqueName: \"kubernetes.io/projected/663eb5d7-3331-4b7b-874f-1eba2aab60ef-kube-api-access-7n57w\") pod \"663eb5d7-3331-4b7b-874f-1eba2aab60ef\" (UID: \"663eb5d7-3331-4b7b-874f-1eba2aab60ef\") " Oct 30 16:31:02.395244 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:02.395087 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/663eb5d7-3331-4b7b-874f-1eba2aab60ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/663eb5d7-3331-4b7b-874f-1eba2aab60ef-metrics-certs\") pod \"663eb5d7-3331-4b7b-874f-1eba2aab60ef\" (UID: \"663eb5d7-3331-4b7b-874f-1eba2aab60ef\") " Oct 30 16:31:02.397115 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:02.397085 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663eb5d7-3331-4b7b-874f-1eba2aab60ef-kube-api-access-7n57w" pod "663eb5d7-3331-4b7b-874f-1eba2aab60ef" (UID: "663eb5d7-3331-4b7b-874f-1eba2aab60ef"). InnerVolumeSpecName "kube-api-access-7n57w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:31:02.397311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:02.397282 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/663eb5d7-3331-4b7b-874f-1eba2aab60ef-metrics-certs" pod "663eb5d7-3331-4b7b-874f-1eba2aab60ef" (UID: "663eb5d7-3331-4b7b-874f-1eba2aab60ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:31:02.495764 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:02.495736 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n57w\" (UniqueName: \"kubernetes.io/projected/663eb5d7-3331-4b7b-874f-1eba2aab60ef-kube-api-access-7n57w\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:31:02.495878 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:02.495770 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/663eb5d7-3331-4b7b-874f-1eba2aab60ef-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:31:02.992396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:02.992357 4861 generic.go:356] "Generic (PLEG): container finished" podID="663eb5d7-3331-4b7b-874f-1eba2aab60ef" containerID="2642314857cbfa2c34f8b9cbf44dbf6ec1e07cabf9ab3d3e77ff2c71ebf94c64" exitCode=0 Oct 30 16:31:02.992794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:02.992405 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" Oct 30 16:31:02.992794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:02.992433 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" event={"ID":"663eb5d7-3331-4b7b-874f-1eba2aab60ef","Type":"ContainerDied","Data":"2642314857cbfa2c34f8b9cbf44dbf6ec1e07cabf9ab3d3e77ff2c71ebf94c64"} Oct 30 16:31:02.992794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:02.992483 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-2w4zs" event={"ID":"663eb5d7-3331-4b7b-874f-1eba2aab60ef","Type":"ContainerDied","Data":"9af8ffec70ef2656e1ae9eb606acc9c60a6122da6a9662a2edb2bc157d73436e"} Oct 30 16:31:02.992794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:02.992505 4861 scope.go:122] "RemoveContainer" containerID="2642314857cbfa2c34f8b9cbf44dbf6ec1e07cabf9ab3d3e77ff2c71ebf94c64" Oct 30 16:31:03.001848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:03.001831 4861 scope.go:122] "RemoveContainer" containerID="2642314857cbfa2c34f8b9cbf44dbf6ec1e07cabf9ab3d3e77ff2c71ebf94c64" Oct 30 16:31:03.002128 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:31:03.002111 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2642314857cbfa2c34f8b9cbf44dbf6ec1e07cabf9ab3d3e77ff2c71ebf94c64\": container with ID starting with 2642314857cbfa2c34f8b9cbf44dbf6ec1e07cabf9ab3d3e77ff2c71ebf94c64 not found: ID does not exist" containerID="2642314857cbfa2c34f8b9cbf44dbf6ec1e07cabf9ab3d3e77ff2c71ebf94c64" Oct 30 16:31:03.002172 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:03.002135 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2642314857cbfa2c34f8b9cbf44dbf6ec1e07cabf9ab3d3e77ff2c71ebf94c64"} err="failed to get container status \"2642314857cbfa2c34f8b9cbf44dbf6ec1e07cabf9ab3d3e77ff2c71ebf94c64\": rpc error: code = NotFound desc = could not find container \"2642314857cbfa2c34f8b9cbf44dbf6ec1e07cabf9ab3d3e77ff2c71ebf94c64\": container with ID starting with 2642314857cbfa2c34f8b9cbf44dbf6ec1e07cabf9ab3d3e77ff2c71ebf94c64 not found: ID does not exist" Oct 30 16:31:03.045376 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:03.045355 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2w4zs"] Oct 30 16:31:03.058125 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:03.058106 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-2w4zs"] Oct 30 16:31:03.661360 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:03.661333 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:03.805351 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:03.805320 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="663eb5d7-3331-4b7b-874f-1eba2aab60ef" path="/var/lib/kubelet/pods/663eb5d7-3331-4b7b-874f-1eba2aab60ef/volumes" Oct 30 16:31:04.159281 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:04.159250 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:05.335276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:05.335250 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:05.474309 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:05.474281 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:09.136433 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:09.136400 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:12.384011 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:12.383977 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:13.649798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.649762 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67"] Oct 30 16:31:13.650180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.650125 4861 cpu_manager.go:414] "containerMap: removing container" podUID="663eb5d7-3331-4b7b-874f-1eba2aab60ef" containerName="manager" Oct 30 16:31:13.650180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.650135 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="663eb5d7-3331-4b7b-874f-1eba2aab60ef" containerName="manager" Oct 30 16:31:13.650258 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.650190 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="663eb5d7-3331-4b7b-874f-1eba2aab60ef" containerName="manager" Oct 30 16:31:13.658250 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.658223 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.674954 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.674931 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67"] Oct 30 16:31:13.685716 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.685695 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.685853 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.685774 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.685922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.685854 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.685922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.685870 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 16:31:13.685922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.685883 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2lpf\" (UniqueName: \"kubernetes.io/projected/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-kube-api-access-q2lpf\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.686044 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.685969 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.686044 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.685999 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.686129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.686104 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 16:31:13.688654 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.688638 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-b826n\"" Oct 30 16:31:13.688756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.688670 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 16:31:13.734762 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.734734 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:13.786625 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.786604 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.786751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.786647 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.786751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.786692 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.786751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.786717 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2lpf\" (UniqueName: \"kubernetes.io/projected/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-kube-api-access-q2lpf\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.786919 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.786765 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.786919 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.786798 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.787163 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.787139 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.787282 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.787201 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.787282 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.787237 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.789051 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.789028 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.789468 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.789448 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.811189 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.811166 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2lpf\" (UniqueName: \"kubernetes.io/projected/c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2-kube-api-access-q2lpf\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-w9k67\" (UID: \"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:13.971524 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:13.971498 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:14.274361 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:14.274333 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67"] Oct 30 16:31:14.275853 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:31:14.275823 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c9cc23_e6a9_42e8_ae3f_f3b3cdd1eac2.slice/crio-54099f72e87ebba910e84024b8e8feda82ad5810f70e8e3e7a34dfc00618a536 WatchSource:0}: Error finding container 54099f72e87ebba910e84024b8e8feda82ad5810f70e8e3e7a34dfc00618a536: Status 404 returned error can't find the container with id 54099f72e87ebba910e84024b8e8feda82ad5810f70e8e3e7a34dfc00618a536 Oct 30 16:31:15.048899 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.048853 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" event={"ID":"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2","Type":"ContainerStarted","Data":"54099f72e87ebba910e84024b8e8feda82ad5810f70e8e3e7a34dfc00618a536"} Oct 30 16:31:15.607212 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.607139 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd"] Oct 30 16:31:15.610651 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.610631 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.643406 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.643378 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 16:31:15.645372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.645349 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd"] Oct 30 16:31:15.694364 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.694331 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:15.702471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.702434 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.702586 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.702488 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztkcn\" (UniqueName: \"kubernetes.io/projected/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-kube-api-access-ztkcn\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.702586 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.702557 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.702662 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.702620 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.702662 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.702643 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.702746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.702717 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.803682 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.803656 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.803848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.803705 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztkcn\" (UniqueName: \"kubernetes.io/projected/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-kube-api-access-ztkcn\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.803848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.803749 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.803848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.803778 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.803848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.803795 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.804056 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.803848 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.804175 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.804150 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.804251 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.804197 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.804251 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.804210 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.806338 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.806313 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.806461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.806445 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.830334 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.830315 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztkcn\" (UniqueName: \"kubernetes.io/projected/b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471-kube-api-access-ztkcn\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-q6hpd\" (UID: \"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:15.925211 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:15.925156 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:16.058769 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:16.057782 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" event={"ID":"c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2","Type":"ContainerStarted","Data":"9cb27bbaa5ef6a6f5d9f61ad5dd3b48319b7197c6fbef9593befce1c3b7a3294"} Oct 30 16:31:16.240605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:16.240546 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" podStartSLOduration=2.20965968 podStartE2EDuration="3.240529558s" totalImagesPullingTime="1.030869878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:16.104629435 +0000 UTC m=+506.447303472" watchObservedRunningTime="2026-10-30 16:31:16.240529558 +0000 UTC m=+506.583203595" Oct 30 16:31:16.241770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:16.241748 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd"] Oct 30 16:31:16.244481 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:31:16.244454 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60d99a0_8ad6_4fc1_aaeb_8e77fe8e6471.slice/crio-f4a3efd0767e5cbe49cde2303c1c8e1b24aa4a1d2ea37ed7e08a93f44a40330c WatchSource:0}: Error finding container f4a3efd0767e5cbe49cde2303c1c8e1b24aa4a1d2ea37ed7e08a93f44a40330c: Status 404 returned error can't find the container with id f4a3efd0767e5cbe49cde2303c1c8e1b24aa4a1d2ea37ed7e08a93f44a40330c Oct 30 16:31:16.971983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:16.971952 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:17.065911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:17.065871 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" event={"ID":"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471","Type":"ContainerStarted","Data":"f26ef849860d01577b21b9013ed90645bc64ecbc1d09fc6b9927602ce30d5488"} Oct 30 16:31:17.066299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:17.065929 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" event={"ID":"b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471","Type":"ContainerStarted","Data":"f4a3efd0767e5cbe49cde2303c1c8e1b24aa4a1d2ea37ed7e08a93f44a40330c"} Oct 30 16:31:17.111146 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:17.111093 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" podStartSLOduration=1.9160763809999999 podStartE2EDuration="2.111079207s" totalImagesPullingTime="195.002826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:17.110205934 +0000 UTC m=+507.452879974" watchObservedRunningTime="2026-10-30 16:31:17.111079207 +0000 UTC m=+507.453753244" Oct 30 16:31:17.925948 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:17.925902 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:18.382645 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:18.382610 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:20.288414 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:31:20.288381 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:31:20.850490 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:20.850457 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-w9pz8"] Oct 30 16:31:20.854785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:20.854761 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:20.875207 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:20.875184 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-w9pz8"] Oct 30 16:31:20.885320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:20.885301 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 16:31:20.941428 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:20.941395 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjgtm\" (UniqueName: \"kubernetes.io/projected/ce7dda96-cefe-44b0-a922-3a1f706dd174-kube-api-access-hjgtm\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:20.941577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:20.941458 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce7dda96-cefe-44b0-a922-3a1f706dd174-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:20.941577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:20.941546 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce7dda96-cefe-44b0-a922-3a1f706dd174-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:20.941647 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:20.941584 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce7dda96-cefe-44b0-a922-3a1f706dd174-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:20.941647 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:20.941614 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce7dda96-cefe-44b0-a922-3a1f706dd174-home\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:20.941647 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:20.941637 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce7dda96-cefe-44b0-a922-3a1f706dd174-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:20.959486 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:20.959462 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:21.042279 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:21.042246 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce7dda96-cefe-44b0-a922-3a1f706dd174-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:21.042418 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:21.042299 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce7dda96-cefe-44b0-a922-3a1f706dd174-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:21.042418 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:21.042354 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce7dda96-cefe-44b0-a922-3a1f706dd174-home\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:21.042418 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:21.042384 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce7dda96-cefe-44b0-a922-3a1f706dd174-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:21.042418 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:21.042417 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjgtm\" (UniqueName: \"kubernetes.io/projected/ce7dda96-cefe-44b0-a922-3a1f706dd174-kube-api-access-hjgtm\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:21.042606 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:21.042471 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce7dda96-cefe-44b0-a922-3a1f706dd174-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:21.042676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:21.042658 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce7dda96-cefe-44b0-a922-3a1f706dd174-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:21.042810 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:21.042787 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce7dda96-cefe-44b0-a922-3a1f706dd174-home\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:21.042900 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:21.042884 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce7dda96-cefe-44b0-a922-3a1f706dd174-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:21.044956 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:21.044924 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce7dda96-cefe-44b0-a922-3a1f706dd174-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:21.045147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:21.045128 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce7dda96-cefe-44b0-a922-3a1f706dd174-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:21.068964 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:21.068942 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjgtm\" (UniqueName: \"kubernetes.io/projected/ce7dda96-cefe-44b0-a922-3a1f706dd174-kube-api-access-hjgtm\") pod \"echo-simulator-kserve-64b6f8f96c-w9pz8\" (UID: \"ce7dda96-cefe-44b0-a922-3a1f706dd174\") " pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:21.168257 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:21.168196 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:21.468001 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:21.467978 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-w9pz8"] Oct 30 16:31:21.469714 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:31:21.469689 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7dda96_cefe_44b0_a922_3a1f706dd174.slice/crio-3a1efb1e6a45320fbaf83bae6ab04bddee3041b12a8e019e8218a4d8fc47bf02 WatchSource:0}: Error finding container 3a1efb1e6a45320fbaf83bae6ab04bddee3041b12a8e019e8218a4d8fc47bf02: Status 404 returned error can't find the container with id 3a1efb1e6a45320fbaf83bae6ab04bddee3041b12a8e019e8218a4d8fc47bf02 Oct 30 16:31:22.091251 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:22.091213 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" event={"ID":"ce7dda96-cefe-44b0-a922-3a1f706dd174","Type":"ContainerStarted","Data":"3a1efb1e6a45320fbaf83bae6ab04bddee3041b12a8e019e8218a4d8fc47bf02"} Oct 30 16:31:23.972125 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:23.972087 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:23.984571 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:23.984548 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:24.105154 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:24.105123 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" event={"ID":"ce7dda96-cefe-44b0-a922-3a1f706dd174","Type":"ContainerStarted","Data":"1fb7e26ae18d68e81f2b2f224dc97cc58e37f337610fe2a1ee11911f94c3e1cc"} Oct 30 16:31:24.116372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:24.116347 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-w9k67" Oct 30 16:31:24.149637 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:24.149591 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" podStartSLOduration=2.51527729 podStartE2EDuration="4.149578083s" totalImagesPullingTime="1.634300793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:24.148353525 +0000 UTC m=+514.491027563" watchObservedRunningTime="2026-10-30 16:31:24.149578083 +0000 UTC m=+514.492252117" Oct 30 16:31:24.168548 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:24.168526 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:25.925660 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:25.925623 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:25.938577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:25.938548 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:26.128088 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:26.128059 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-q6hpd" Oct 30 16:31:26.133203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:26.133178 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:27.707237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.707202 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd"] Oct 30 16:31:27.710983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.710959 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.727647 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.727624 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 16:31:27.742867 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.742841 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd"] Oct 30 16:31:27.798450 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.798421 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:27.809303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.809277 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37ba69c-1fa5-4f9b-a754-80057e2608b4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.809412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.809313 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37ba69c-1fa5-4f9b-a754-80057e2608b4-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.809412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.809328 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37ba69c-1fa5-4f9b-a754-80057e2608b4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.809412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.809351 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37ba69c-1fa5-4f9b-a754-80057e2608b4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.809534 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.809447 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8n9p\" (UniqueName: \"kubernetes.io/projected/d37ba69c-1fa5-4f9b-a754-80057e2608b4-kube-api-access-w8n9p\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.809534 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.809476 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37ba69c-1fa5-4f9b-a754-80057e2608b4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.910534 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.910504 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37ba69c-1fa5-4f9b-a754-80057e2608b4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.910671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.910543 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37ba69c-1fa5-4f9b-a754-80057e2608b4-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.910671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.910561 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37ba69c-1fa5-4f9b-a754-80057e2608b4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.910671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.910594 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37ba69c-1fa5-4f9b-a754-80057e2608b4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.910671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.910638 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8n9p\" (UniqueName: \"kubernetes.io/projected/d37ba69c-1fa5-4f9b-a754-80057e2608b4-kube-api-access-w8n9p\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.910906 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.910679 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37ba69c-1fa5-4f9b-a754-80057e2608b4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.911024 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.911000 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37ba69c-1fa5-4f9b-a754-80057e2608b4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.911102 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.911059 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37ba69c-1fa5-4f9b-a754-80057e2608b4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.911102 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.911084 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37ba69c-1fa5-4f9b-a754-80057e2608b4-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.913141 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.913119 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37ba69c-1fa5-4f9b-a754-80057e2608b4-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.913430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.913414 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37ba69c-1fa5-4f9b-a754-80057e2608b4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:27.935144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:27.935115 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8n9p\" (UniqueName: \"kubernetes.io/projected/d37ba69c-1fa5-4f9b-a754-80057e2608b4-kube-api-access-w8n9p\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd\" (UID: \"d37ba69c-1fa5-4f9b-a754-80057e2608b4\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:28.037553 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:28.037529 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:28.343102 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:28.343080 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd"] Oct 30 16:31:28.345259 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:31:28.345234 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd37ba69c_1fa5_4f9b_a754_80057e2608b4.slice/crio-1b1540fe61d7d4dd0fa24a2494c8b7268252e6fe975df51aa2e13a2709e4c8f6 WatchSource:0}: Error finding container 1b1540fe61d7d4dd0fa24a2494c8b7268252e6fe975df51aa2e13a2709e4c8f6: Status 404 returned error can't find the container with id 1b1540fe61d7d4dd0fa24a2494c8b7268252e6fe975df51aa2e13a2709e4c8f6 Oct 30 16:31:29.131314 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:29.131281 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" event={"ID":"d37ba69c-1fa5-4f9b-a754-80057e2608b4","Type":"ContainerStarted","Data":"1dcc208ac12805d9f20698f34d75bbd78c74e04e34239ba820e696d15f2fbf9e"} Oct 30 16:31:29.131676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:29.131322 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" event={"ID":"d37ba69c-1fa5-4f9b-a754-80057e2608b4","Type":"ContainerStarted","Data":"1b1540fe61d7d4dd0fa24a2494c8b7268252e6fe975df51aa2e13a2709e4c8f6"} Oct 30 16:31:29.180943 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:29.180900 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podStartSLOduration=1.963661249 podStartE2EDuration="2.180886431s" totalImagesPullingTime="217.225182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:29.179235764 +0000 UTC m=+519.521909803" watchObservedRunningTime="2026-10-30 16:31:29.180886431 +0000 UTC m=+519.523560471" Oct 30 16:31:30.038217 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:30.038187 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:31.168740 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:31.168698 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:31.180938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:31.180905 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:32.155661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:32.155633 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-w9pz8" Oct 30 16:31:38.037964 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:38.037929 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:38.050512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:38.050486 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:38.181926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:38.181902 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:31:40.800405 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:40.800371 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:41.305816 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.305782 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m"] Oct 30 16:31:41.309223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.309199 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.320836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.320815 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd"] Oct 30 16:31:41.323926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.323910 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.331944 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.331924 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xtm6x\"" Oct 30 16:31:41.335301 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.335272 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m"] Oct 30 16:31:41.348180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.348146 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd"] Oct 30 16:31:41.428632 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.428605 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67a6d47-cf04-4307-af5b-ecb10e0612b1-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.428817 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.428636 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.428817 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.428664 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67a6d47-cf04-4307-af5b-ecb10e0612b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.428817 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.428754 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.428817 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.428783 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57crd\" (UniqueName: \"kubernetes.io/projected/c2cb80d7-4995-4584-8781-0c5921d6115c-kube-api-access-57crd\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.429033 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.428831 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67a6d47-cf04-4307-af5b-ecb10e0612b1-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.429033 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.428847 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.429033 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.428875 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67a6d47-cf04-4307-af5b-ecb10e0612b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.429033 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.428902 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.429033 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.428966 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.429033 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.428983 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2cb80d7-4995-4584-8781-0c5921d6115c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.429307 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.429101 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67a6d47-cf04-4307-af5b-ecb10e0612b1-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.429307 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.429162 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67a6d47-cf04-4307-af5b-ecb10e0612b1-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.429307 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.429196 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w6s8\" (UniqueName: \"kubernetes.io/projected/f67a6d47-cf04-4307-af5b-ecb10e0612b1-kube-api-access-8w6s8\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.429307 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.429225 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.429307 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.429270 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67a6d47-cf04-4307-af5b-ecb10e0612b1-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.429307 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.429290 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67a6d47-cf04-4307-af5b-ecb10e0612b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.429508 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.429321 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.530124 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.530088 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.530124 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.530131 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67a6d47-cf04-4307-af5b-ecb10e0612b1-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.530361 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.530156 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67a6d47-cf04-4307-af5b-ecb10e0612b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.530361 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.530182 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.530361 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.530209 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67a6d47-cf04-4307-af5b-ecb10e0612b1-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.530585 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.530561 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.530670 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.530590 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67a6d47-cf04-4307-af5b-ecb10e0612b1-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.530670 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.530592 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.530670 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.530629 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.530965 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.530914 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67a6d47-cf04-4307-af5b-ecb10e0612b1-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.531309 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.531282 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67a6d47-cf04-4307-af5b-ecb10e0612b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.531406 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.531326 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.531406 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.531351 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57crd\" (UniqueName: \"kubernetes.io/projected/c2cb80d7-4995-4584-8781-0c5921d6115c-kube-api-access-57crd\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.531406 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.531384 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67a6d47-cf04-4307-af5b-ecb10e0612b1-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.531575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.531409 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.531575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.531443 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67a6d47-cf04-4307-af5b-ecb10e0612b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.531575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.531482 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.531575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.531539 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.532046 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.531567 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2cb80d7-4995-4584-8781-0c5921d6115c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.532225 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.532207 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67a6d47-cf04-4307-af5b-ecb10e0612b1-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.532302 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.532275 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67a6d47-cf04-4307-af5b-ecb10e0612b1-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.532357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.532307 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w6s8\" (UniqueName: \"kubernetes.io/projected/f67a6d47-cf04-4307-af5b-ecb10e0612b1-kube-api-access-8w6s8\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.532468 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.532447 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67a6d47-cf04-4307-af5b-ecb10e0612b1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.532906 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.532886 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67a6d47-cf04-4307-af5b-ecb10e0612b1-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.533082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.533054 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2cb80d7-4995-4584-8781-0c5921d6115c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.533163 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.533114 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67a6d47-cf04-4307-af5b-ecb10e0612b1-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.533550 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.533500 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.533849 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.533816 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.535276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.535252 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67a6d47-cf04-4307-af5b-ecb10e0612b1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.536123 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.536101 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67a6d47-cf04-4307-af5b-ecb10e0612b1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.536232 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.536144 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.536455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.536432 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.555913 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.555848 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57crd\" (UniqueName: \"kubernetes.io/projected/c2cb80d7-4995-4584-8781-0c5921d6115c-kube-api-access-57crd\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.556317 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.556298 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-cfqjd\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.556421 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.556399 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w6s8\" (UniqueName: \"kubernetes.io/projected/f67a6d47-cf04-4307-af5b-ecb10e0612b1-kube-api-access-8w6s8\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.558009 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.557987 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67a6d47-cf04-4307-af5b-ecb10e0612b1-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-smq4m\" (UID: \"f67a6d47-cf04-4307-af5b-ecb10e0612b1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.623488 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.623460 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:41.638879 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.638852 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:41.962003 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.961974 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd"] Oct 30 16:31:41.963523 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:31:41.963487 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2cb80d7_4995_4584_8781_0c5921d6115c.slice/crio-29a4f9f614d5ba680c341cada252c5bacc5ae33f4d7f6e5c1c97a671df89be96 WatchSource:0}: Error finding container 29a4f9f614d5ba680c341cada252c5bacc5ae33f4d7f6e5c1c97a671df89be96: Status 404 returned error can't find the container with id 29a4f9f614d5ba680c341cada252c5bacc5ae33f4d7f6e5c1c97a671df89be96 Oct 30 16:31:41.968226 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:41.968206 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m"] Oct 30 16:31:41.976493 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:31:41.976471 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67a6d47_cf04_4307_af5b_ecb10e0612b1.slice/crio-a00fc78b8cfbfce5f6c816a098b035c4e525489505b26e0073619e426c01add9 WatchSource:0}: Error finding container a00fc78b8cfbfce5f6c816a098b035c4e525489505b26e0073619e426c01add9: Status 404 returned error can't find the container with id a00fc78b8cfbfce5f6c816a098b035c4e525489505b26e0073619e426c01add9 Oct 30 16:31:42.191915 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:42.191877 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" event={"ID":"c2cb80d7-4995-4584-8781-0c5921d6115c","Type":"ContainerStarted","Data":"29a4f9f614d5ba680c341cada252c5bacc5ae33f4d7f6e5c1c97a671df89be96"} Oct 30 16:31:42.193074 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:42.193052 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" event={"ID":"f67a6d47-cf04-4307-af5b-ecb10e0612b1","Type":"ContainerStarted","Data":"a00fc78b8cfbfce5f6c816a098b035c4e525489505b26e0073619e426c01add9"} Oct 30 16:31:44.597840 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:44.597811 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:31:44.598245 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:44.597889 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:31:44.598245 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:44.597935 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:31:45.158941 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:45.158900 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:31:45.159053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:45.158976 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:31:45.159053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:45.159006 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:31:45.235700 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:45.235662 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" event={"ID":"f67a6d47-cf04-4307-af5b-ecb10e0612b1","Type":"ContainerStarted","Data":"578ef66f1ab470f7f71edfd35460d123bfd7c4a44909f25a3f706eaf0cc76f34"} Oct 30 16:31:45.296623 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:45.296567 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" podStartSLOduration=1.6778968779999999 podStartE2EDuration="4.296552745s" totalImagesPullingTime="2.618655867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:45.294836914 +0000 UTC m=+535.637510955" watchObservedRunningTime="2026-10-30 16:31:45.296552745 +0000 UTC m=+535.639226783" Oct 30 16:31:45.624115 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:45.624085 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:45.626299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:45.626256 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-smq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:31:45.626447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:45.626327 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" podUID="f67a6d47-cf04-4307-af5b-ecb10e0612b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 16:31:46.255847 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:46.255799 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" event={"ID":"c2cb80d7-4995-4584-8781-0c5921d6115c","Type":"ContainerStarted","Data":"02f3bff03b8d3ec09a0cd116b78411a940ff1b088487ee61bde855dc8c0786eb"} Oct 30 16:31:46.627975 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:46.627887 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-smq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:31:46.628543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:46.628513 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" podUID="f67a6d47-cf04-4307-af5b-ecb10e0612b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 16:31:46.643647 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:46.643619 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:46.643985 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:46.643963 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-cfqjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:31:46.644162 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:46.644140 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" podUID="c2cb80d7-4995-4584-8781-0c5921d6115c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 16:31:47.633142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:47.633102 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-smq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:31:47.633780 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:47.633689 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" podUID="f67a6d47-cf04-4307-af5b-ecb10e0612b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 16:31:47.643161 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:47.643066 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-cfqjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:31:47.643161 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:47.643111 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" podUID="c2cb80d7-4995-4584-8781-0c5921d6115c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 16:31:47.819440 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:47.819405 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:47.964701 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:47.964622 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:48.624101 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:48.624065 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-smq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:31:48.624372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:48.624344 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" podUID="f67a6d47-cf04-4307-af5b-ecb10e0612b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 16:31:48.646747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:48.646058 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-cfqjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:31:48.646747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:48.646107 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" podUID="c2cb80d7-4995-4584-8781-0c5921d6115c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 16:31:49.299004 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:49.298963 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:49.489350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:49.489300 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:31:49.624174 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:49.624086 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-smq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:31:49.624433 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:49.624404 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" podUID="f67a6d47-cf04-4307-af5b-ecb10e0612b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 16:31:49.642065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:49.642036 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-cfqjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:31:49.642285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:49.642259 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" podUID="c2cb80d7-4995-4584-8781-0c5921d6115c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 16:31:50.290626 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:31:50.290595 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:31:50.624807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:50.624111 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-smq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:31:50.624807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:50.624173 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" podUID="f67a6d47-cf04-4307-af5b-ecb10e0612b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 16:31:50.639981 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:50.639949 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-cfqjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:31:50.640128 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:50.639999 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" podUID="c2cb80d7-4995-4584-8781-0c5921d6115c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 16:31:51.623874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:51.623837 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-smq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Oct 30 16:31:51.624311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:51.623931 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" podUID="f67a6d47-cf04-4307-af5b-ecb10e0612b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Oct 30 16:31:51.639764 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:51.639715 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-cfqjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:31:51.639858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:51.639790 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" podUID="c2cb80d7-4995-4584-8781-0c5921d6115c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 16:31:52.640250 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:52.640209 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-cfqjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:31:52.640641 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:52.640277 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" podUID="c2cb80d7-4995-4584-8781-0c5921d6115c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Oct 30 16:31:52.961192 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:52.960901 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:52.961421 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:52.961398 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:52.962132 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:52.962110 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" Oct 30 16:31:53.022926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:53.022875 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" podStartSLOduration=8.830519496 podStartE2EDuration="12.022863536s" totalImagesPullingTime="3.19234404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:46.327341385 +0000 UTC m=+536.670015424" watchObservedRunningTime="2026-10-30 16:31:53.022863536 +0000 UTC m=+543.365537574" Oct 30 16:31:53.644009 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:53.643977 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:53.644440 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:53.644251 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:31:53.644886 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:31:53.644868 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:32:14.577591 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:14.577558 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:14.744511 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:14.744476 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:14.891498 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:14.891429 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:15.083509 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.083480 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:15.268567 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.268538 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:15.427469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.427430 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:15.547541 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.547468 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl"] Oct 30 16:32:15.551670 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.551650 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.575512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.575481 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl"] Oct 30 16:32:15.583630 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.583600 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:15.584013 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.583819 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b1c7fd-dba6-422b-bd28-92702748babc-istio-podinfo\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.584013 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.583855 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-workload-certs\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.584013 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.583873 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-istio-envoy\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.584138 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.584037 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-credential-socket\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.584138 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.584060 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b1c7fd-dba6-422b-bd28-92702748babc-istio-token\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.584138 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.584082 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-workload-socket\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.584138 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.584110 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jcgj\" (UniqueName: \"kubernetes.io/projected/21b1c7fd-dba6-422b-bd28-92702748babc-kube-api-access-9jcgj\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.584334 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.584264 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-istio-data\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.584334 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.584308 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b1c7fd-dba6-422b-bd28-92702748babc-istiod-ca-cert\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.589596 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.589576 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-37964af0-gw-openshift-default-dockercfg-qfg42\"" Oct 30 16:32:15.685039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.685011 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-workload-certs\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.685039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.685042 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-istio-envoy\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.685447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.685082 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-credential-socket\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.685447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.685373 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b1c7fd-dba6-422b-bd28-92702748babc-istio-token\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.685447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.685435 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-workload-socket\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.685609 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.685500 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jcgj\" (UniqueName: \"kubernetes.io/projected/21b1c7fd-dba6-422b-bd28-92702748babc-kube-api-access-9jcgj\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.685609 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.685532 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-workload-certs\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.685609 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.685551 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-credential-socket\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.685609 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.685592 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-istio-data\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.685927 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.685638 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b1c7fd-dba6-422b-bd28-92702748babc-istiod-ca-cert\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.685927 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.685697 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b1c7fd-dba6-422b-bd28-92702748babc-istio-podinfo\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.686056 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.685940 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-istio-data\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.686209 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.686191 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-workload-socket\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.686372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.686349 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b1c7fd-dba6-422b-bd28-92702748babc-istiod-ca-cert\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.688367 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.688301 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-istio-envoy\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.704698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.704671 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b1c7fd-dba6-422b-bd28-92702748babc-istio-podinfo\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.711079 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.711049 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b1c7fd-dba6-422b-bd28-92702748babc-istio-token\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.711204 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.711106 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jcgj\" (UniqueName: \"kubernetes.io/projected/21b1c7fd-dba6-422b-bd28-92702748babc-kube-api-access-9jcgj\") pod \"e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.729589 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.729564 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:15.865264 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.865185 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:15.879186 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:15.879166 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:16.185315 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.185292 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl"] Oct 30 16:32:16.186905 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:32:16.186876 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b1c7fd_dba6_422b_bd28_92702748babc.slice/crio-f7175cf7bae8cf043b6039b32c74498c9e23dd339db8948604addb0a897b63f1 WatchSource:0}: Error finding container f7175cf7bae8cf043b6039b32c74498c9e23dd339db8948604addb0a897b63f1: Status 404 returned error can't find the container with id f7175cf7bae8cf043b6039b32c74498c9e23dd339db8948604addb0a897b63f1 Oct 30 16:32:16.189088 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.189058 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:16.189190 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.189132 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:16.189259 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.189202 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:16.400309 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.400280 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:16.408434 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.408406 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" event={"ID":"21b1c7fd-dba6-422b-bd28-92702748babc","Type":"ContainerStarted","Data":"d4ebf7ef297e8bed45e853d7bf78d498deb39e5f3451abdb0e78482a234bee85"} Oct 30 16:32:16.408582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.408461 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" event={"ID":"21b1c7fd-dba6-422b-bd28-92702748babc","Type":"ContainerStarted","Data":"f7175cf7bae8cf043b6039b32c74498c9e23dd339db8948604addb0a897b63f1"} Oct 30 16:32:16.456508 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.456402 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" podStartSLOduration=1.456386535 podStartE2EDuration="1.456386535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:16.455456134 +0000 UTC m=+566.798130172" watchObservedRunningTime="2026-10-30 16:32:16.456386535 +0000 UTC m=+566.799060639" Oct 30 16:32:16.785362 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.785321 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck"] Oct 30 16:32:16.793163 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.793139 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:16.810418 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.810389 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a2d47c16-openshift-default-dockercfg-v4hkq\"" Oct 30 16:32:16.813616 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.813587 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck"] Oct 30 16:32:16.865933 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.865911 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:16.885489 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.885462 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:16.901114 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.901069 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-workload-socket\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:16.901307 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.901130 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-istio-data\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:16.901391 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.901358 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-credential-socket\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:16.901479 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.901463 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734244c1-0481-45e6-a479-5af996594d19-istiod-ca-cert\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:16.901532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.901509 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734244c1-0481-45e6-a479-5af996594d19-istio-podinfo\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:16.901610 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.901591 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-workload-certs\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:16.901705 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.901687 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t87fd\" (UniqueName: \"kubernetes.io/projected/734244c1-0481-45e6-a479-5af996594d19-kube-api-access-t87fd\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:16.901818 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.901780 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-istio-envoy\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:16.901952 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:16.901851 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/734244c1-0481-45e6-a479-5af996594d19-istio-token\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.002784 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.002749 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t87fd\" (UniqueName: \"kubernetes.io/projected/734244c1-0481-45e6-a479-5af996594d19-kube-api-access-t87fd\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.002784 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.002784 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-istio-envoy\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.003009 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.002808 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/734244c1-0481-45e6-a479-5af996594d19-istio-token\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.003009 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.002938 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-workload-socket\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.003009 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.002993 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-istio-data\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.003173 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.003030 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-credential-socket\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.003173 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.003088 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734244c1-0481-45e6-a479-5af996594d19-istiod-ca-cert\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.003173 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.003123 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734244c1-0481-45e6-a479-5af996594d19-istio-podinfo\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.003334 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.003271 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-workload-certs\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.003394 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.003358 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-workload-socket\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.003576 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.003552 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-credential-socket\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.003576 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.003567 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-istio-data\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.003758 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.003739 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-workload-certs\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.004150 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.004129 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734244c1-0481-45e6-a479-5af996594d19-istiod-ca-cert\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.005416 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.005398 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-istio-envoy\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.005735 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.005706 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734244c1-0481-45e6-a479-5af996594d19-istio-podinfo\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.026894 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.026870 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/734244c1-0481-45e6-a479-5af996594d19-istio-token\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.027058 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.027038 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t87fd\" (UniqueName: \"kubernetes.io/projected/734244c1-0481-45e6-a479-5af996594d19-kube-api-access-t87fd\") pod \"e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.059012 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.058953 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:17.122037 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.121516 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:17.458074 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.458051 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck"] Oct 30 16:32:17.459798 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:32:17.459772 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734244c1_0481_45e6_a479_5af996594d19.slice/crio-3befaeb6b41cb6a2b77bf44cb92e231c1bf3afdb6d8cb9a1979a4aa0108cf588 WatchSource:0}: Error finding container 3befaeb6b41cb6a2b77bf44cb92e231c1bf3afdb6d8cb9a1979a4aa0108cf588: Status 404 returned error can't find the container with id 3befaeb6b41cb6a2b77bf44cb92e231c1bf3afdb6d8cb9a1979a4aa0108cf588 Oct 30 16:32:17.462109 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.462082 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:17.462184 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.462138 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:17.462184 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.462166 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:17.871146 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:17.871117 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:18.383752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:18.383694 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:18.420435 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:18.420401 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" event={"ID":"734244c1-0481-45e6-a479-5af996594d19","Type":"ContainerStarted","Data":"e516133f5120d47a1d1795df515b2bcbe1873f7d73c68d8551582164eda05f99"} Oct 30 16:32:18.420603 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:18.420501 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" event={"ID":"734244c1-0481-45e6-a479-5af996594d19","Type":"ContainerStarted","Data":"3befaeb6b41cb6a2b77bf44cb92e231c1bf3afdb6d8cb9a1979a4aa0108cf588"} Oct 30 16:32:18.420670 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:18.420626 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:18.421540 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:18.421524 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:18.491956 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:18.491906 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" podStartSLOduration=2.491895002 podStartE2EDuration="2.491895002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:18.479419863 +0000 UTC m=+568.822093902" watchObservedRunningTime="2026-10-30 16:32:18.491895002 +0000 UTC m=+568.834569039" Oct 30 16:32:18.540944 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:18.540915 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:18.941551 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:18.941502 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:19.105408 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.105373 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:19.123070 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.123039 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:19.128421 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.128397 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:19.310225 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.310188 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:19.426082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.426012 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:19.427154 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.427125 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:19.467224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.467191 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp"] Oct 30 16:32:19.471209 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.471193 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" Oct 30 16:32:19.478843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.478818 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:19.490398 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.490369 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-37964af0\"" Oct 30 16:32:19.498759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.496166 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-37964af0-dockercfg-lsb9n\"" Oct 30 16:32:19.498759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.497173 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp"] Oct 30 16:32:19.547057 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.547023 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c3cd45-bf72-4b25-83d1-42a01952a1cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp\" (UID: \"22c3cd45-bf72-4b25-83d1-42a01952a1cf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" Oct 30 16:32:19.551930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.547301 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5npv5\" (UniqueName: \"kubernetes.io/projected/22c3cd45-bf72-4b25-83d1-42a01952a1cf-kube-api-access-5npv5\") pod \"payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp\" (UID: \"22c3cd45-bf72-4b25-83d1-42a01952a1cf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" Oct 30 16:32:19.562123 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.562062 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h"] Oct 30 16:32:19.565958 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.565938 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" Oct 30 16:32:19.591960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.591925 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h"] Oct 30 16:32:19.649263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.648469 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl8df\" (UniqueName: \"kubernetes.io/projected/d2197c23-4f38-406b-80fc-57daeb425e15-kube-api-access-hl8df\") pod \"payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h\" (UID: \"d2197c23-4f38-406b-80fc-57daeb425e15\") " pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" Oct 30 16:32:19.649263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.648521 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5npv5\" (UniqueName: \"kubernetes.io/projected/22c3cd45-bf72-4b25-83d1-42a01952a1cf-kube-api-access-5npv5\") pod \"payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp\" (UID: \"22c3cd45-bf72-4b25-83d1-42a01952a1cf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" Oct 30 16:32:19.649263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.648598 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2197c23-4f38-406b-80fc-57daeb425e15-plugins-config-volume\") pod \"payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h\" (UID: \"d2197c23-4f38-406b-80fc-57daeb425e15\") " pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" Oct 30 16:32:19.649263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.648658 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c3cd45-bf72-4b25-83d1-42a01952a1cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp\" (UID: \"22c3cd45-bf72-4b25-83d1-42a01952a1cf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" Oct 30 16:32:19.649582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.649558 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c3cd45-bf72-4b25-83d1-42a01952a1cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp\" (UID: \"22c3cd45-bf72-4b25-83d1-42a01952a1cf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" Oct 30 16:32:19.673605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.673578 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5npv5\" (UniqueName: \"kubernetes.io/projected/22c3cd45-bf72-4b25-83d1-42a01952a1cf-kube-api-access-5npv5\") pod \"payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp\" (UID: \"22c3cd45-bf72-4b25-83d1-42a01952a1cf\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" Oct 30 16:32:19.686627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.686598 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:19.749438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.749403 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl8df\" (UniqueName: \"kubernetes.io/projected/d2197c23-4f38-406b-80fc-57daeb425e15-kube-api-access-hl8df\") pod \"payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h\" (UID: \"d2197c23-4f38-406b-80fc-57daeb425e15\") " pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" Oct 30 16:32:19.749596 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.749504 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2197c23-4f38-406b-80fc-57daeb425e15-plugins-config-volume\") pod \"payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h\" (UID: \"d2197c23-4f38-406b-80fc-57daeb425e15\") " pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" Oct 30 16:32:19.750191 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.750174 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2197c23-4f38-406b-80fc-57daeb425e15-plugins-config-volume\") pod \"payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h\" (UID: \"d2197c23-4f38-406b-80fc-57daeb425e15\") " pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" Oct 30 16:32:19.773572 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.773544 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl8df\" (UniqueName: \"kubernetes.io/projected/d2197c23-4f38-406b-80fc-57daeb425e15-kube-api-access-hl8df\") pod \"payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h\" (UID: \"d2197c23-4f38-406b-80fc-57daeb425e15\") " pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" Oct 30 16:32:19.784829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.784802 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" Oct 30 16:32:19.886107 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:19.886078 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" Oct 30 16:32:20.059286 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.059261 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs"] Oct 30 16:32:20.064197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.064179 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:20.083691 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.083649 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-37964af0-serving-cert\"" Oct 30 16:32:20.085459 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.085431 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs"] Oct 30 16:32:20.102399 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.102364 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp"] Oct 30 16:32:20.105409 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:32:20.105375 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c3cd45_bf72_4b25_83d1_42a01952a1cf.slice/crio-4c69fc337f72a4dc7cf5c22a788ecc60a14214166e5643606f9bfa40514d8a5d WatchSource:0}: Error finding container 4c69fc337f72a4dc7cf5c22a788ecc60a14214166e5643606f9bfa40514d8a5d: Status 404 returned error can't find the container with id 4c69fc337f72a4dc7cf5c22a788ecc60a14214166e5643606f9bfa40514d8a5d Oct 30 16:32:20.157134 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.157106 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldrfp\" (UniqueName: \"kubernetes.io/projected/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-kube-api-access-ldrfp\") pod \"maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs\" (UID: \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:20.157265 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.157201 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-openshift-service-ca\") pod \"maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs\" (UID: \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:20.157397 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.157374 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-metrics-certs\") pod \"maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs\" (UID: \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:20.157463 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.157446 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-maas-api-tls\") pod \"maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs\" (UID: \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:20.185656 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.185635 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:20.197920 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.197895 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h"] Oct 30 16:32:20.199103 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:32:20.199078 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2197c23_4f38_406b_80fc_57daeb425e15.slice/crio-be2898c3f084e17bb0c1918a1fab7cd2ce0c0e6891fabb3e3bc663f102940df7 WatchSource:0}: Error finding container be2898c3f084e17bb0c1918a1fab7cd2ce0c0e6891fabb3e3bc663f102940df7: Status 404 returned error can't find the container with id be2898c3f084e17bb0c1918a1fab7cd2ce0c0e6891fabb3e3bc663f102940df7 Oct 30 16:32:20.258306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.258275 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-metrics-certs\") pod \"maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs\" (UID: \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:20.258430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.258328 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-maas-api-tls\") pod \"maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs\" (UID: \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:20.258430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.258356 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldrfp\" (UniqueName: \"kubernetes.io/projected/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-kube-api-access-ldrfp\") pod \"maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs\" (UID: \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:20.258430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.258424 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-openshift-service-ca\") pod \"maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs\" (UID: \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:20.259112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.259092 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-openshift-service-ca\") pod \"maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs\" (UID: \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:20.260936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.260912 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-maas-api-tls\") pod \"maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs\" (UID: \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:20.261351 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.261331 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-metrics-certs\") pod \"maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs\" (UID: \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:20.297347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.297308 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldrfp\" (UniqueName: \"kubernetes.io/projected/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-kube-api-access-ldrfp\") pod \"maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs\" (UID: \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:20.317557 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:32:20.317523 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:32:20.377807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.377732 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:20.385806 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.385777 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:20.440313 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.440280 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" event={"ID":"d2197c23-4f38-406b-80fc-57daeb425e15","Type":"ContainerStarted","Data":"81cc55ae044ff76c786c474e5fc4afa5c0d9b31cb7b6e4fec4126fea13dc00a6"} Oct 30 16:32:20.440452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.440341 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" event={"ID":"d2197c23-4f38-406b-80fc-57daeb425e15","Type":"ContainerStarted","Data":"be2898c3f084e17bb0c1918a1fab7cd2ce0c0e6891fabb3e3bc663f102940df7"} Oct 30 16:32:20.440452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.440375 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" Oct 30 16:32:20.444619 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.444457 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" event={"ID":"22c3cd45-bf72-4b25-83d1-42a01952a1cf","Type":"ContainerStarted","Data":"d7b88dbd7db0c6355f206e0f532a19268aa676e93e75ee5c5aa78906e9e3f3ba"} Oct 30 16:32:20.444619 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.444503 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" event={"ID":"22c3cd45-bf72-4b25-83d1-42a01952a1cf","Type":"ContainerStarted","Data":"4c69fc337f72a4dc7cf5c22a788ecc60a14214166e5643606f9bfa40514d8a5d"} Oct 30 16:32:20.445190 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.445163 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" Oct 30 16:32:20.484803 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.484757 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" podStartSLOduration=1.484712309 podStartE2EDuration="1.484712309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:20.483865605 +0000 UTC m=+570.826539645" watchObservedRunningTime="2026-10-30 16:32:20.484712309 +0000 UTC m=+570.827386346" Oct 30 16:32:20.525504 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.525461 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" podStartSLOduration=1.52545043 podStartE2EDuration="1.52545043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:20.525134167 +0000 UTC m=+570.867808205" watchObservedRunningTime="2026-10-30 16:32:20.52545043 +0000 UTC m=+570.868124464" Oct 30 16:32:20.704907 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:20.704881 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs"] Oct 30 16:32:20.706512 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:32:20.706479 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3bfc9f9_c2fd_443e_b4d3_2c5982b8f479.slice/crio-ba6a973db37614fc9b1c94c943c290e7071441eb542e009715cbdee3be2ea027 WatchSource:0}: Error finding container ba6a973db37614fc9b1c94c943c290e7071441eb542e009715cbdee3be2ea027: Status 404 returned error can't find the container with id ba6a973db37614fc9b1c94c943c290e7071441eb542e009715cbdee3be2ea027 Oct 30 16:32:21.042150 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:21.042115 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:21.195067 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:21.195024 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:21.468134 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:21.468082 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" event={"ID":"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479","Type":"ContainerStarted","Data":"a77d000f653ea6da97da1d1a1fa84f65682e4f3e0857858568189865e7991b03"} Oct 30 16:32:21.468334 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:21.468197 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" event={"ID":"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479","Type":"ContainerStarted","Data":"ba6a973db37614fc9b1c94c943c290e7071441eb542e009715cbdee3be2ea027"} Oct 30 16:32:21.468897 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:21.468700 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:21.515185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:21.515132 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" podStartSLOduration=1.024354975 podStartE2EDuration="1.515118979s" totalImagesPullingTime="490.764004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:21.513019795 +0000 UTC m=+571.855693832" watchObservedRunningTime="2026-10-30 16:32:21.515118979 +0000 UTC m=+571.857793017" Oct 30 16:32:21.515910 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:21.515875 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:21.684953 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:21.684866 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:21.977874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:21.977844 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:22.160455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.160426 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:22.173547 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.173366 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn"] Oct 30 16:32:22.179105 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.179085 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:22.203136 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.203039 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn"] Oct 30 16:32:22.237141 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.237044 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f756b7bf-serving-cert\"" Oct 30 16:32:22.286345 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.286316 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edd32141-41dd-4b75-996e-f3c9fc03a396-metrics-certs\") pod \"maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn\" (UID: \"edd32141-41dd-4b75-996e-f3c9fc03a396\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:22.286513 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.286358 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfxqj\" (UniqueName: \"kubernetes.io/projected/edd32141-41dd-4b75-996e-f3c9fc03a396-kube-api-access-hfxqj\") pod \"maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn\" (UID: \"edd32141-41dd-4b75-996e-f3c9fc03a396\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:22.286513 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.286379 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edd32141-41dd-4b75-996e-f3c9fc03a396-maas-api-tls\") pod \"maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn\" (UID: \"edd32141-41dd-4b75-996e-f3c9fc03a396\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:22.286651 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.286544 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd32141-41dd-4b75-996e-f3c9fc03a396-openshift-service-ca\") pod \"maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn\" (UID: \"edd32141-41dd-4b75-996e-f3c9fc03a396\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:22.336133 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.336097 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:22.387311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.387272 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edd32141-41dd-4b75-996e-f3c9fc03a396-metrics-certs\") pod \"maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn\" (UID: \"edd32141-41dd-4b75-996e-f3c9fc03a396\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:22.387481 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.387325 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfxqj\" (UniqueName: \"kubernetes.io/projected/edd32141-41dd-4b75-996e-f3c9fc03a396-kube-api-access-hfxqj\") pod \"maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn\" (UID: \"edd32141-41dd-4b75-996e-f3c9fc03a396\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:22.387481 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.387357 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edd32141-41dd-4b75-996e-f3c9fc03a396-maas-api-tls\") pod \"maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn\" (UID: \"edd32141-41dd-4b75-996e-f3c9fc03a396\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:22.387481 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.387421 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd32141-41dd-4b75-996e-f3c9fc03a396-openshift-service-ca\") pod \"maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn\" (UID: \"edd32141-41dd-4b75-996e-f3c9fc03a396\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:22.388090 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.388065 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd32141-41dd-4b75-996e-f3c9fc03a396-openshift-service-ca\") pod \"maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn\" (UID: \"edd32141-41dd-4b75-996e-f3c9fc03a396\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:22.389823 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.389794 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edd32141-41dd-4b75-996e-f3c9fc03a396-metrics-certs\") pod \"maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn\" (UID: \"edd32141-41dd-4b75-996e-f3c9fc03a396\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:22.389960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.389884 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edd32141-41dd-4b75-996e-f3c9fc03a396-maas-api-tls\") pod \"maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn\" (UID: \"edd32141-41dd-4b75-996e-f3c9fc03a396\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:22.413349 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.413321 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfxqj\" (UniqueName: \"kubernetes.io/projected/edd32141-41dd-4b75-996e-f3c9fc03a396-kube-api-access-hfxqj\") pod \"maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn\" (UID: \"edd32141-41dd-4b75-996e-f3c9fc03a396\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:22.484912 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.484880 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:22.525692 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.525618 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:22.861560 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:22.861537 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn"] Oct 30 16:32:22.863317 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:32:22.863295 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd32141_41dd_4b75_996e_f3c9fc03a396.slice/crio-8e13d39efc07e184087654a60727428ccd2c9a70b45089ac29dee0c489c6693e WatchSource:0}: Error finding container 8e13d39efc07e184087654a60727428ccd2c9a70b45089ac29dee0c489c6693e: Status 404 returned error can't find the container with id 8e13d39efc07e184087654a60727428ccd2c9a70b45089ac29dee0c489c6693e Oct 30 16:32:23.485204 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:23.485163 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" event={"ID":"edd32141-41dd-4b75-996e-f3c9fc03a396","Type":"ContainerStarted","Data":"89d0cf323d413d244560183c2bedb01f22d458afc598a81661329a1ecf40aaec"} Oct 30 16:32:23.485624 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:23.485222 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" event={"ID":"edd32141-41dd-4b75-996e-f3c9fc03a396","Type":"ContainerStarted","Data":"8e13d39efc07e184087654a60727428ccd2c9a70b45089ac29dee0c489c6693e"} Oct 30 16:32:23.485624 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:23.485411 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:23.531880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:23.531779 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" podStartSLOduration=1.134181758 podStartE2EDuration="1.531765189s" totalImagesPullingTime="397.583431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:23.529621011 +0000 UTC m=+573.872295048" watchObservedRunningTime="2026-10-30 16:32:23.531765189 +0000 UTC m=+573.874439221" Oct 30 16:32:23.629348 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:23.629249 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:23.782520 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:23.782447 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:23.923749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:23.923130 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:24.397426 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.397397 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7"] Oct 30 16:32:24.419749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.406975 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:24.434768 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.424473 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7"] Oct 30 16:32:24.440576 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.440550 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:24.451958 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.451935 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a2d47c16-serving-cert\"" Oct 30 16:32:24.550813 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.550783 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-maas-api-tls\") pod \"maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:24.551337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.550853 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-metrics-certs\") pod \"maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:24.551337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.550907 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww5zs\" (UniqueName: \"kubernetes.io/projected/7c6f5108-546f-4a8f-93f4-960cf5838f69-kube-api-access-ww5zs\") pod \"maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:24.551337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.551031 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6f5108-546f-4a8f-93f4-960cf5838f69-openshift-service-ca\") pod \"maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:24.585668 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.585640 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:24.652575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.652505 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-maas-api-tls\") pod \"maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:24.652764 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.652581 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-metrics-certs\") pod \"maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:24.652764 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.652609 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww5zs\" (UniqueName: \"kubernetes.io/projected/7c6f5108-546f-4a8f-93f4-960cf5838f69-kube-api-access-ww5zs\") pod \"maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:24.652764 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:32:24.652664 4861 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-serving-cert: secret "maas-api-e2e-ait-a2d47c16-serving-cert" not found Oct 30 16:32:24.652949 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:32:24.652780 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-maas-api-tls podName:7c6f5108-546f-4a8f-93f4-960cf5838f69 nodeName:}" failed. No retries permitted until 2026-10-30 16:32:25.15275507 +0000 UTC m=+575.495429108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-maas-api-tls") pod "maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" (UID: "7c6f5108-546f-4a8f-93f4-960cf5838f69") : secret "maas-api-e2e-ait-a2d47c16-serving-cert" not found Oct 30 16:32:24.653071 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.652692 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6f5108-546f-4a8f-93f4-960cf5838f69-openshift-service-ca\") pod \"maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:24.653596 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.653567 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6f5108-546f-4a8f-93f4-960cf5838f69-openshift-service-ca\") pod \"maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:24.655248 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.655227 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-metrics-certs\") pod \"maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:24.678021 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.677995 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww5zs\" (UniqueName: \"kubernetes.io/projected/7c6f5108-546f-4a8f-93f4-960cf5838f69-kube-api-access-ww5zs\") pod \"maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:24.752453 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.752376 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:24.897187 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:24.897149 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:25.161337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:25.161298 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-maas-api-tls\") pod \"maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:25.164756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:25.164708 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-maas-api-tls\") pod \"maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:25.352870 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:25.352833 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:25.677273 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:25.677248 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7"] Oct 30 16:32:25.678583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:32:25.678552 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6f5108_546f_4a8f_93f4_960cf5838f69.slice/crio-f115d9566ff774cd5fd249d61dff19b18837eb36af7052faa4d1d2a3d0bda9aa WatchSource:0}: Error finding container f115d9566ff774cd5fd249d61dff19b18837eb36af7052faa4d1d2a3d0bda9aa: Status 404 returned error can't find the container with id f115d9566ff774cd5fd249d61dff19b18837eb36af7052faa4d1d2a3d0bda9aa Oct 30 16:32:26.521747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.520659 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" event={"ID":"7c6f5108-546f-4a8f-93f4-960cf5838f69","Type":"ContainerStarted","Data":"a38d0e4facb38d81486f4e023568e31ff5daa995546b99c7f59efc3924aedd46"} Oct 30 16:32:26.521747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.520757 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" event={"ID":"7c6f5108-546f-4a8f-93f4-960cf5838f69","Type":"ContainerStarted","Data":"f115d9566ff774cd5fd249d61dff19b18837eb36af7052faa4d1d2a3d0bda9aa"} Oct 30 16:32:26.521747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.521674 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:26.554404 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.554365 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9"] Oct 30 16:32:26.558178 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.558161 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:26.570251 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.570225 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9"] Oct 30 16:32:26.571130 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.571086 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" podStartSLOduration=2.2522342269999998 podStartE2EDuration="2.57107452s" totalImagesPullingTime="318.840293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:26.570496848 +0000 UTC m=+576.913170885" watchObservedRunningTime="2026-10-30 16:32:26.57107452 +0000 UTC m=+576.913748558" Oct 30 16:32:26.573271 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.573252 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-cff59996-serving-cert\"" Oct 30 16:32:26.676825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.676796 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0181dec-1f2a-4d48-b71a-5ce318ae042a-metrics-certs\") pod \"maas-api-e2e-ait-cff59996-547cd54966-9ffs9\" (UID: \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:26.677018 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.676876 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0181dec-1f2a-4d48-b71a-5ce318ae042a-openshift-service-ca\") pod \"maas-api-e2e-ait-cff59996-547cd54966-9ffs9\" (UID: \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:26.677018 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.676912 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvc7b\" (UniqueName: \"kubernetes.io/projected/c0181dec-1f2a-4d48-b71a-5ce318ae042a-kube-api-access-zvc7b\") pod \"maas-api-e2e-ait-cff59996-547cd54966-9ffs9\" (UID: \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:26.677131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.677049 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0181dec-1f2a-4d48-b71a-5ce318ae042a-maas-api-tls\") pod \"maas-api-e2e-ait-cff59996-547cd54966-9ffs9\" (UID: \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:26.690652 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.690628 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:26.777751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.777637 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0181dec-1f2a-4d48-b71a-5ce318ae042a-maas-api-tls\") pod \"maas-api-e2e-ait-cff59996-547cd54966-9ffs9\" (UID: \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:26.777751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.777701 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0181dec-1f2a-4d48-b71a-5ce318ae042a-metrics-certs\") pod \"maas-api-e2e-ait-cff59996-547cd54966-9ffs9\" (UID: \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:26.777954 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.777777 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0181dec-1f2a-4d48-b71a-5ce318ae042a-openshift-service-ca\") pod \"maas-api-e2e-ait-cff59996-547cd54966-9ffs9\" (UID: \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:26.777954 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.777811 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvc7b\" (UniqueName: \"kubernetes.io/projected/c0181dec-1f2a-4d48-b71a-5ce318ae042a-kube-api-access-zvc7b\") pod \"maas-api-e2e-ait-cff59996-547cd54966-9ffs9\" (UID: \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:26.778511 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.778483 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0181dec-1f2a-4d48-b71a-5ce318ae042a-openshift-service-ca\") pod \"maas-api-e2e-ait-cff59996-547cd54966-9ffs9\" (UID: \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:26.780324 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.780303 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0181dec-1f2a-4d48-b71a-5ce318ae042a-maas-api-tls\") pod \"maas-api-e2e-ait-cff59996-547cd54966-9ffs9\" (UID: \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:26.780641 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.780343 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0181dec-1f2a-4d48-b71a-5ce318ae042a-metrics-certs\") pod \"maas-api-e2e-ait-cff59996-547cd54966-9ffs9\" (UID: \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:26.814744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.801187 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvc7b\" (UniqueName: \"kubernetes.io/projected/c0181dec-1f2a-4d48-b71a-5ce318ae042a-kube-api-access-zvc7b\") pod \"maas-api-e2e-ait-cff59996-547cd54966-9ffs9\" (UID: \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:26.872931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.872901 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:26.875861 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:26.875833 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:27.222071 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:27.220571 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9"] Oct 30 16:32:27.226791 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:32:27.226261 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0181dec_1f2a_4d48_b71a_5ce318ae042a.slice/crio-8b7eb9811852bfeb93c9bdcac600378ec00e889375d77e03446bda63e341d24f WatchSource:0}: Error finding container 8b7eb9811852bfeb93c9bdcac600378ec00e889375d77e03446bda63e341d24f: Status 404 returned error can't find the container with id 8b7eb9811852bfeb93c9bdcac600378ec00e889375d77e03446bda63e341d24f Oct 30 16:32:27.530135 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:27.530049 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" event={"ID":"c0181dec-1f2a-4d48-b71a-5ce318ae042a","Type":"ContainerStarted","Data":"8b7eb9811852bfeb93c9bdcac600378ec00e889375d77e03446bda63e341d24f"} Oct 30 16:32:28.139919 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:28.139882 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:28.297691 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:28.297656 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:28.491123 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:28.491096 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:28.539987 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:28.539953 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" event={"ID":"c0181dec-1f2a-4d48-b71a-5ce318ae042a","Type":"ContainerStarted","Data":"c612b617fb5fcae2fe11ce36de5b051208bf2ce535429af15d8c9d89bc482ecb"} Oct 30 16:32:28.540134 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:28.540028 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:28.637304 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:28.637243 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" podStartSLOduration=2.310165528 podStartE2EDuration="2.637228393s" totalImagesPullingTime="327.062865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:28.601887281 +0000 UTC m=+578.944561332" watchObservedRunningTime="2026-10-30 16:32:28.637228393 +0000 UTC m=+578.979902430" Oct 30 16:32:29.158063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:29.158028 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:29.498529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:29.498501 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:29.984709 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:29.984673 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:30.162671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:30.162640 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:31.225867 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:31.224145 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:31.471904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:31.471871 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" Oct 30 16:32:32.481213 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:32.481187 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" Oct 30 16:32:33.146452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:33.146293 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:33.422808 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:33.422713 4861 patch_prober.go:29] interesting pod/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 16:32:33.422808 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:33.422796 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" podUID="21b1c7fd-dba6-422b-bd28-92702748babc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 16:32:33.545665 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:33.545641 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:33.570101 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:33.570074 4861 generic.go:356] "Generic (PLEG): container finished" podID="21b1c7fd-dba6-422b-bd28-92702748babc" containerID="d4ebf7ef297e8bed45e853d7bf78d498deb39e5f3451abdb0e78482a234bee85" exitCode=0 Oct 30 16:32:33.570262 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:33.570148 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" event={"ID":"21b1c7fd-dba6-422b-bd28-92702748babc","Type":"ContainerDied","Data":"d4ebf7ef297e8bed45e853d7bf78d498deb39e5f3451abdb0e78482a234bee85"} Oct 30 16:32:33.570532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:33.570513 4861 scope.go:122] "RemoveContainer" containerID="d4ebf7ef297e8bed45e853d7bf78d498deb39e5f3451abdb0e78482a234bee85" Oct 30 16:32:33.571757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:33.571707 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:33.571855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:33.571800 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:33.571855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:33.571834 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:34.551580 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:34.551541 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:34.578870 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:34.578837 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" event={"ID":"21b1c7fd-dba6-422b-bd28-92702748babc","Type":"ContainerStarted","Data":"aa4a892bf499f16e577b511b0b9db869c2a5dfcbf03d751786af885bcb246195"} Oct 30 16:32:34.865869 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:34.865778 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:34.866036 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:34.865994 4861 patch_prober.go:29] interesting pod/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 16:32:34.866084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:34.866036 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" podUID="21b1c7fd-dba6-422b-bd28-92702748babc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 16:32:35.240282 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:35.237708 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:35.468886 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:35.468843 4861 patch_prober.go:29] interesting pod/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:32:35.469075 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:35.468911 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" podUID="734244c1-0481-45e6-a479-5af996594d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:32:35.865643 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:35.865604 4861 patch_prober.go:29] interesting pod/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 16:32:35.866056 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:35.865665 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" podUID="21b1c7fd-dba6-422b-bd28-92702748babc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 16:32:36.866531 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:36.866501 4861 patch_prober.go:29] interesting pod/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 16:32:36.866918 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:36.866554 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" podUID="21b1c7fd-dba6-422b-bd28-92702748babc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 16:32:37.329523 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:37.328926 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:37.562862 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:37.562778 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:37.596518 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:37.596487 4861 generic.go:356] "Generic (PLEG): container finished" podID="734244c1-0481-45e6-a479-5af996594d19" containerID="e516133f5120d47a1d1795df515b2bcbe1873f7d73c68d8551582164eda05f99" exitCode=0 Oct 30 16:32:37.596671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:37.596560 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" event={"ID":"734244c1-0481-45e6-a479-5af996594d19","Type":"ContainerDied","Data":"e516133f5120d47a1d1795df515b2bcbe1873f7d73c68d8551582164eda05f99"} Oct 30 16:32:37.597000 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:37.596986 4861 scope.go:122] "RemoveContainer" containerID="e516133f5120d47a1d1795df515b2bcbe1873f7d73c68d8551582164eda05f99" Oct 30 16:32:37.598232 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:37.598209 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:37.598286 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:37.598274 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:37.598322 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:37.598304 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:37.865674 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:37.865586 4861 patch_prober.go:29] interesting pod/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 16:32:37.865674 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:37.865655 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" podUID="21b1c7fd-dba6-422b-bd28-92702748babc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 16:32:38.653908 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:38.653871 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" event={"ID":"734244c1-0481-45e6-a479-5af996594d19","Type":"ContainerStarted","Data":"3f8bd6ced7c2dfd7e186818f69ff780298637df04cc677afcf199120a1cb8f90"} Oct 30 16:32:38.866323 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:38.866284 4861 patch_prober.go:29] interesting pod/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 16:32:38.866501 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:38.866334 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" podUID="21b1c7fd-dba6-422b-bd28-92702748babc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 16:32:39.130752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:39.127067 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:39.130752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:39.127209 4861 patch_prober.go:29] interesting pod/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 16:32:39.130752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:39.127249 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" podUID="734244c1-0481-45e6-a479-5af996594d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 16:32:39.513770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:39.513270 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:39.870389 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:39.870315 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:39.870771 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:39.870684 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:39.871368 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:39.871348 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:32:40.125805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:40.125092 4861 patch_prober.go:29] interesting pod/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 16:32:40.125805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:40.125165 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" podUID="734244c1-0481-45e6-a479-5af996594d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 16:32:40.984572 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:40.984543 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:41.128237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:41.128201 4861 patch_prober.go:29] interesting pod/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 16:32:41.128584 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:41.128541 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" podUID="734244c1-0481-45e6-a479-5af996594d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 16:32:41.727956 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:41.727921 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:42.125183 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:42.125095 4861 patch_prober.go:29] interesting pod/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 16:32:42.125759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:42.125703 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" podUID="734244c1-0481-45e6-a479-5af996594d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 16:32:43.127063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:43.127016 4861 patch_prober.go:29] interesting pod/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Oct 30 16:32:43.127464 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:43.127077 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" podUID="734244c1-0481-45e6-a479-5af996594d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Oct 30 16:32:43.758752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:43.758172 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:44.128379 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:44.128342 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:44.128834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:44.128815 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:44.129362 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:44.129343 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:32:45.606987 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:45.606947 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:47.591133 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:47.591101 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:48.813903 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:48.813868 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h"] Oct 30 16:32:48.849252 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:48.849222 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp"] Oct 30 16:32:48.987572 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:48.987546 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:49.139120 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:49.139037 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:49.577077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:49.577037 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:49.854471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:49.854378 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:49.861242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:49.861219 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:32:49.861988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:49.861968 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:32:50.014585 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:50.014553 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:50.304326 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:32:50.304287 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:32:51.444248 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:51.444178 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:52.016222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:52.016188 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:52.172592 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:52.172567 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:53.449933 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:53.449895 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:53.814916 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:53.814883 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" podUID="d2197c23-4f38-406b-80fc-57daeb425e15" containerName="payload-processing" containerID="cri-o://81cc55ae044ff76c786c474e5fc4afa5c0d9b31cb7b6e4fec4126fea13dc00a6" gracePeriod=25 Oct 30 16:32:53.850342 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:53.850306 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" podUID="22c3cd45-bf72-4b25-83d1-42a01952a1cf" containerName="payload-pre-processing" containerID="cri-o://d7b88dbd7db0c6355f206e0f532a19268aa676e93e75ee5c5aa78906e9e3f3ba" gracePeriod=25 Oct 30 16:32:54.105499 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.105469 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" Oct 30 16:32:54.140029 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.140007 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" Oct 30 16:32:54.267091 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.267047 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2197c23-4f38-406b-80fc-57daeb425e15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2197c23-4f38-406b-80fc-57daeb425e15-plugins-config-volume\") pod \"d2197c23-4f38-406b-80fc-57daeb425e15\" (UID: \"d2197c23-4f38-406b-80fc-57daeb425e15\") " Oct 30 16:32:54.267283 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.267154 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c3cd45-bf72-4b25-83d1-42a01952a1cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c3cd45-bf72-4b25-83d1-42a01952a1cf-plugins-config-volume\") pod \"22c3cd45-bf72-4b25-83d1-42a01952a1cf\" (UID: \"22c3cd45-bf72-4b25-83d1-42a01952a1cf\") " Oct 30 16:32:54.267283 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.267241 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2197c23-4f38-406b-80fc-57daeb425e15-kube-api-access-hl8df\" (UniqueName: \"kubernetes.io/projected/d2197c23-4f38-406b-80fc-57daeb425e15-kube-api-access-hl8df\") pod \"d2197c23-4f38-406b-80fc-57daeb425e15\" (UID: \"d2197c23-4f38-406b-80fc-57daeb425e15\") " Oct 30 16:32:54.267419 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.267336 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c3cd45-bf72-4b25-83d1-42a01952a1cf-kube-api-access-5npv5\" (UniqueName: \"kubernetes.io/projected/22c3cd45-bf72-4b25-83d1-42a01952a1cf-kube-api-access-5npv5\") pod \"22c3cd45-bf72-4b25-83d1-42a01952a1cf\" (UID: \"22c3cd45-bf72-4b25-83d1-42a01952a1cf\") " Oct 30 16:32:54.267529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.267503 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2197c23-4f38-406b-80fc-57daeb425e15-plugins-config-volume" pod "d2197c23-4f38-406b-80fc-57daeb425e15" (UID: "d2197c23-4f38-406b-80fc-57daeb425e15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:32:54.267598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.267562 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c3cd45-bf72-4b25-83d1-42a01952a1cf-plugins-config-volume" pod "22c3cd45-bf72-4b25-83d1-42a01952a1cf" (UID: "22c3cd45-bf72-4b25-83d1-42a01952a1cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:32:54.267696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.267683 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2197c23-4f38-406b-80fc-57daeb425e15-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:54.267776 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.267699 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22c3cd45-bf72-4b25-83d1-42a01952a1cf-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:54.269467 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.269449 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2197c23-4f38-406b-80fc-57daeb425e15-kube-api-access-hl8df" pod "d2197c23-4f38-406b-80fc-57daeb425e15" (UID: "d2197c23-4f38-406b-80fc-57daeb425e15"). InnerVolumeSpecName "kube-api-access-hl8df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:32:54.269467 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.269454 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c3cd45-bf72-4b25-83d1-42a01952a1cf-kube-api-access-5npv5" pod "22c3cd45-bf72-4b25-83d1-42a01952a1cf" (UID: "22c3cd45-bf72-4b25-83d1-42a01952a1cf"). InnerVolumeSpecName "kube-api-access-5npv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:32:54.369210 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.369134 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl8df\" (UniqueName: \"kubernetes.io/projected/d2197c23-4f38-406b-80fc-57daeb425e15-kube-api-access-hl8df\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:54.369210 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.369158 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5npv5\" (UniqueName: \"kubernetes.io/projected/22c3cd45-bf72-4b25-83d1-42a01952a1cf-kube-api-access-5npv5\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:54.738929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.738891 4861 generic.go:356] "Generic (PLEG): container finished" podID="d2197c23-4f38-406b-80fc-57daeb425e15" containerID="81cc55ae044ff76c786c474e5fc4afa5c0d9b31cb7b6e4fec4126fea13dc00a6" exitCode=0 Oct 30 16:32:54.739401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.738996 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" event={"ID":"d2197c23-4f38-406b-80fc-57daeb425e15","Type":"ContainerDied","Data":"81cc55ae044ff76c786c474e5fc4afa5c0d9b31cb7b6e4fec4126fea13dc00a6"} Oct 30 16:32:54.739401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.739005 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" Oct 30 16:32:54.739401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.739088 4861 scope.go:122] "RemoveContainer" containerID="81cc55ae044ff76c786c474e5fc4afa5c0d9b31cb7b6e4fec4126fea13dc00a6" Oct 30 16:32:54.739401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.739076 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h" event={"ID":"d2197c23-4f38-406b-80fc-57daeb425e15","Type":"ContainerDied","Data":"be2898c3f084e17bb0c1918a1fab7cd2ce0c0e6891fabb3e3bc663f102940df7"} Oct 30 16:32:54.740367 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.740345 4861 generic.go:356] "Generic (PLEG): container finished" podID="22c3cd45-bf72-4b25-83d1-42a01952a1cf" containerID="d7b88dbd7db0c6355f206e0f532a19268aa676e93e75ee5c5aa78906e9e3f3ba" exitCode=0 Oct 30 16:32:54.740461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.740388 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" event={"ID":"22c3cd45-bf72-4b25-83d1-42a01952a1cf","Type":"ContainerDied","Data":"d7b88dbd7db0c6355f206e0f532a19268aa676e93e75ee5c5aa78906e9e3f3ba"} Oct 30 16:32:54.740461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.740401 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" Oct 30 16:32:54.740461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.740424 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp" event={"ID":"22c3cd45-bf72-4b25-83d1-42a01952a1cf","Type":"ContainerDied","Data":"4c69fc337f72a4dc7cf5c22a788ecc60a14214166e5643606f9bfa40514d8a5d"} Oct 30 16:32:54.749988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.749971 4861 scope.go:122] "RemoveContainer" containerID="81cc55ae044ff76c786c474e5fc4afa5c0d9b31cb7b6e4fec4126fea13dc00a6" Oct 30 16:32:54.750342 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:32:54.750313 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81cc55ae044ff76c786c474e5fc4afa5c0d9b31cb7b6e4fec4126fea13dc00a6\": container with ID starting with 81cc55ae044ff76c786c474e5fc4afa5c0d9b31cb7b6e4fec4126fea13dc00a6 not found: ID does not exist" containerID="81cc55ae044ff76c786c474e5fc4afa5c0d9b31cb7b6e4fec4126fea13dc00a6" Oct 30 16:32:54.750455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.750348 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81cc55ae044ff76c786c474e5fc4afa5c0d9b31cb7b6e4fec4126fea13dc00a6"} err="failed to get container status \"81cc55ae044ff76c786c474e5fc4afa5c0d9b31cb7b6e4fec4126fea13dc00a6\": rpc error: code = NotFound desc = could not find container \"81cc55ae044ff76c786c474e5fc4afa5c0d9b31cb7b6e4fec4126fea13dc00a6\": container with ID starting with 81cc55ae044ff76c786c474e5fc4afa5c0d9b31cb7b6e4fec4126fea13dc00a6 not found: ID does not exist" Oct 30 16:32:54.750455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.750371 4861 scope.go:122] "RemoveContainer" containerID="d7b88dbd7db0c6355f206e0f532a19268aa676e93e75ee5c5aa78906e9e3f3ba" Oct 30 16:32:54.758891 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.758875 4861 scope.go:122] "RemoveContainer" containerID="d7b88dbd7db0c6355f206e0f532a19268aa676e93e75ee5c5aa78906e9e3f3ba" Oct 30 16:32:54.759173 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:32:54.759149 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b88dbd7db0c6355f206e0f532a19268aa676e93e75ee5c5aa78906e9e3f3ba\": container with ID starting with d7b88dbd7db0c6355f206e0f532a19268aa676e93e75ee5c5aa78906e9e3f3ba not found: ID does not exist" containerID="d7b88dbd7db0c6355f206e0f532a19268aa676e93e75ee5c5aa78906e9e3f3ba" Oct 30 16:32:54.759267 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.759173 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b88dbd7db0c6355f206e0f532a19268aa676e93e75ee5c5aa78906e9e3f3ba"} err="failed to get container status \"d7b88dbd7db0c6355f206e0f532a19268aa676e93e75ee5c5aa78906e9e3f3ba\": rpc error: code = NotFound desc = could not find container \"d7b88dbd7db0c6355f206e0f532a19268aa676e93e75ee5c5aa78906e9e3f3ba\": container with ID starting with d7b88dbd7db0c6355f206e0f532a19268aa676e93e75ee5c5aa78906e9e3f3ba not found: ID does not exist" Oct 30 16:32:54.792417 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.792394 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h"] Oct 30 16:32:54.805081 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.805061 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-37964af0-85c6cd97fb-w224h"] Oct 30 16:32:54.830794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.830768 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp"] Oct 30 16:32:54.834203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:54.834186 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-37964af0-dd97d6bff-24srp"] Oct 30 16:32:55.037610 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.037534 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs"] Oct 30 16:32:55.037874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.037820 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" podUID="b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479" containerName="maas-api" containerID="cri-o://a77d000f653ea6da97da1d1a1fa84f65682e4f3e0857858568189865e7991b03" gracePeriod=30 Oct 30 16:32:55.119591 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.119556 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:55.307255 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.307230 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:55.426853 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.426820 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn"] Oct 30 16:32:55.427139 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.427100 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" podUID="edd32141-41dd-4b75-996e-f3c9fc03a396" containerName="maas-api" containerID="cri-o://89d0cf323d413d244560183c2bedb01f22d458afc598a81661329a1ecf40aaec" gracePeriod=30 Oct 30 16:32:55.479876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.479845 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-kube-api-access-ldrfp\" (UniqueName: \"kubernetes.io/projected/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-kube-api-access-ldrfp\") pod \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\" (UID: \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\") " Oct 30 16:32:55.480017 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.479897 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-openshift-service-ca\") pod \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\" (UID: \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\") " Oct 30 16:32:55.480017 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.479980 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-maas-api-tls\") pod \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\" (UID: \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\") " Oct 30 16:32:55.480163 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.480042 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-metrics-certs\") pod \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\" (UID: \"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479\") " Oct 30 16:32:55.480331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.480298 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-openshift-service-ca" pod "b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479" (UID: "b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:32:55.482148 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.482126 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-maas-api-tls" pod "b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479" (UID: "b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:32:55.482647 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.482626 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-kube-api-access-ldrfp" pod "b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479" (UID: "b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479"). InnerVolumeSpecName "kube-api-access-ldrfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:32:55.482762 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.482628 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-metrics-certs" pod "b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479" (UID: "b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:32:55.542919 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.542883 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:55.581608 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.581578 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldrfp\" (UniqueName: \"kubernetes.io/projected/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-kube-api-access-ldrfp\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:55.581608 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.581610 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:55.581608 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.581624 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:55.582072 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.581637 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:55.668328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.668300 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:55.695163 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.695113 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:55.751098 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.751068 4861 generic.go:356] "Generic (PLEG): container finished" podID="b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479" containerID="a77d000f653ea6da97da1d1a1fa84f65682e4f3e0857858568189865e7991b03" exitCode=0 Oct 30 16:32:55.751551 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.751120 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" Oct 30 16:32:55.751551 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.751153 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" event={"ID":"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479","Type":"ContainerDied","Data":"a77d000f653ea6da97da1d1a1fa84f65682e4f3e0857858568189865e7991b03"} Oct 30 16:32:55.751551 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.751211 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs" event={"ID":"b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479","Type":"ContainerDied","Data":"ba6a973db37614fc9b1c94c943c290e7071441eb542e009715cbdee3be2ea027"} Oct 30 16:32:55.751551 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.751243 4861 scope.go:122] "RemoveContainer" containerID="a77d000f653ea6da97da1d1a1fa84f65682e4f3e0857858568189865e7991b03" Oct 30 16:32:55.752896 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.752869 4861 generic.go:356] "Generic (PLEG): container finished" podID="edd32141-41dd-4b75-996e-f3c9fc03a396" containerID="89d0cf323d413d244560183c2bedb01f22d458afc598a81661329a1ecf40aaec" exitCode=0 Oct 30 16:32:55.753019 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.752929 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" Oct 30 16:32:55.753068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.752927 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" event={"ID":"edd32141-41dd-4b75-996e-f3c9fc03a396","Type":"ContainerDied","Data":"89d0cf323d413d244560183c2bedb01f22d458afc598a81661329a1ecf40aaec"} Oct 30 16:32:55.753106 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.753077 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn" event={"ID":"edd32141-41dd-4b75-996e-f3c9fc03a396","Type":"ContainerDied","Data":"8e13d39efc07e184087654a60727428ccd2c9a70b45089ac29dee0c489c6693e"} Oct 30 16:32:55.764314 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.764299 4861 scope.go:122] "RemoveContainer" containerID="a77d000f653ea6da97da1d1a1fa84f65682e4f3e0857858568189865e7991b03" Oct 30 16:32:55.764537 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:32:55.764521 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77d000f653ea6da97da1d1a1fa84f65682e4f3e0857858568189865e7991b03\": container with ID starting with a77d000f653ea6da97da1d1a1fa84f65682e4f3e0857858568189865e7991b03 not found: ID does not exist" containerID="a77d000f653ea6da97da1d1a1fa84f65682e4f3e0857858568189865e7991b03" Oct 30 16:32:55.764579 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.764545 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77d000f653ea6da97da1d1a1fa84f65682e4f3e0857858568189865e7991b03"} err="failed to get container status \"a77d000f653ea6da97da1d1a1fa84f65682e4f3e0857858568189865e7991b03\": rpc error: code = NotFound desc = could not find container \"a77d000f653ea6da97da1d1a1fa84f65682e4f3e0857858568189865e7991b03\": container with ID starting with a77d000f653ea6da97da1d1a1fa84f65682e4f3e0857858568189865e7991b03 not found: ID does not exist" Oct 30 16:32:55.764579 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.764563 4861 scope.go:122] "RemoveContainer" containerID="89d0cf323d413d244560183c2bedb01f22d458afc598a81661329a1ecf40aaec" Oct 30 16:32:55.773046 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.773026 4861 scope.go:122] "RemoveContainer" containerID="89d0cf323d413d244560183c2bedb01f22d458afc598a81661329a1ecf40aaec" Oct 30 16:32:55.773349 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:32:55.773314 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d0cf323d413d244560183c2bedb01f22d458afc598a81661329a1ecf40aaec\": container with ID starting with 89d0cf323d413d244560183c2bedb01f22d458afc598a81661329a1ecf40aaec not found: ID does not exist" containerID="89d0cf323d413d244560183c2bedb01f22d458afc598a81661329a1ecf40aaec" Oct 30 16:32:55.773437 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.773355 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d0cf323d413d244560183c2bedb01f22d458afc598a81661329a1ecf40aaec"} err="failed to get container status \"89d0cf323d413d244560183c2bedb01f22d458afc598a81661329a1ecf40aaec\": rpc error: code = NotFound desc = could not find container \"89d0cf323d413d244560183c2bedb01f22d458afc598a81661329a1ecf40aaec\": container with ID starting with 89d0cf323d413d244560183c2bedb01f22d458afc598a81661329a1ecf40aaec not found: ID does not exist" Oct 30 16:32:55.784132 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.784107 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edd32141-41dd-4b75-996e-f3c9fc03a396-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd32141-41dd-4b75-996e-f3c9fc03a396-openshift-service-ca\") pod \"edd32141-41dd-4b75-996e-f3c9fc03a396\" (UID: \"edd32141-41dd-4b75-996e-f3c9fc03a396\") " Oct 30 16:32:55.784237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.784146 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edd32141-41dd-4b75-996e-f3c9fc03a396-kube-api-access-hfxqj\" (UniqueName: \"kubernetes.io/projected/edd32141-41dd-4b75-996e-f3c9fc03a396-kube-api-access-hfxqj\") pod \"edd32141-41dd-4b75-996e-f3c9fc03a396\" (UID: \"edd32141-41dd-4b75-996e-f3c9fc03a396\") " Oct 30 16:32:55.784237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.784207 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edd32141-41dd-4b75-996e-f3c9fc03a396-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edd32141-41dd-4b75-996e-f3c9fc03a396-maas-api-tls\") pod \"edd32141-41dd-4b75-996e-f3c9fc03a396\" (UID: \"edd32141-41dd-4b75-996e-f3c9fc03a396\") " Oct 30 16:32:55.784383 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.784370 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edd32141-41dd-4b75-996e-f3c9fc03a396-metrics-certs\" (UniqueName: \"kubernetes.io/secret/edd32141-41dd-4b75-996e-f3c9fc03a396-metrics-certs\") pod \"edd32141-41dd-4b75-996e-f3c9fc03a396\" (UID: \"edd32141-41dd-4b75-996e-f3c9fc03a396\") " Oct 30 16:32:55.784442 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.784407 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edd32141-41dd-4b75-996e-f3c9fc03a396-openshift-service-ca" pod "edd32141-41dd-4b75-996e-f3c9fc03a396" (UID: "edd32141-41dd-4b75-996e-f3c9fc03a396"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:32:55.784668 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.784651 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edd32141-41dd-4b75-996e-f3c9fc03a396-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:55.786313 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.786293 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edd32141-41dd-4b75-996e-f3c9fc03a396-kube-api-access-hfxqj" pod "edd32141-41dd-4b75-996e-f3c9fc03a396" (UID: "edd32141-41dd-4b75-996e-f3c9fc03a396"). InnerVolumeSpecName "kube-api-access-hfxqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:32:55.786379 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.786355 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edd32141-41dd-4b75-996e-f3c9fc03a396-maas-api-tls" pod "edd32141-41dd-4b75-996e-f3c9fc03a396" (UID: "edd32141-41dd-4b75-996e-f3c9fc03a396"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:32:55.786415 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.786391 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edd32141-41dd-4b75-996e-f3c9fc03a396-metrics-certs" pod "edd32141-41dd-4b75-996e-f3c9fc03a396" (UID: "edd32141-41dd-4b75-996e-f3c9fc03a396"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:32:55.806098 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.806044 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c3cd45-bf72-4b25-83d1-42a01952a1cf" path="/var/lib/kubelet/pods/22c3cd45-bf72-4b25-83d1-42a01952a1cf/volumes" Oct 30 16:32:55.806401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.806388 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2197c23-4f38-406b-80fc-57daeb425e15" path="/var/lib/kubelet/pods/d2197c23-4f38-406b-80fc-57daeb425e15/volumes" Oct 30 16:32:55.806779 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.806765 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs"] Oct 30 16:32:55.811343 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.811320 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-37964af0-845dc4c96d-htkqs"] Oct 30 16:32:55.833878 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.833850 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:55.876993 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.876953 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7"] Oct 30 16:32:55.877205 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.877178 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" podUID="7c6f5108-546f-4a8f-93f4-960cf5838f69" containerName="maas-api" containerID="cri-o://a38d0e4facb38d81486f4e023568e31ff5daa995546b99c7f59efc3924aedd46" gracePeriod=30 Oct 30 16:32:55.886683 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.886173 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edd32141-41dd-4b75-996e-f3c9fc03a396-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:55.886683 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.886233 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfxqj\" (UniqueName: \"kubernetes.io/projected/edd32141-41dd-4b75-996e-f3c9fc03a396-kube-api-access-hfxqj\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:55.886683 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.886248 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edd32141-41dd-4b75-996e-f3c9fc03a396-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:55.929863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.929358 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl"] Oct 30 16:32:55.929863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.929553 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" podUID="21b1c7fd-dba6-422b-bd28-92702748babc" containerName="istio-proxy" containerID="cri-o://aa4a892bf499f16e577b511b0b9db869c2a5dfcbf03d751786af885bcb246195" gracePeriod=30 Oct 30 16:32:55.982594 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:55.982563 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:56.096929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.096838 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn"] Oct 30 16:32:56.101211 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.101187 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f756b7bf-7bb6f6698d-g7rtn"] Oct 30 16:32:56.146760 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.140470 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:56.161514 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.161210 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:56.288957 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.288925 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6f5108-546f-4a8f-93f4-960cf5838f69-kube-api-access-ww5zs\" (UniqueName: \"kubernetes.io/projected/7c6f5108-546f-4a8f-93f4-960cf5838f69-kube-api-access-ww5zs\") pod \"7c6f5108-546f-4a8f-93f4-960cf5838f69\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " Oct 30 16:32:56.289122 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.288967 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-metrics-certs\") pod \"7c6f5108-546f-4a8f-93f4-960cf5838f69\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " Oct 30 16:32:56.289122 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.289074 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-maas-api-tls\") pod \"7c6f5108-546f-4a8f-93f4-960cf5838f69\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " Oct 30 16:32:56.289122 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.289102 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6f5108-546f-4a8f-93f4-960cf5838f69-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6f5108-546f-4a8f-93f4-960cf5838f69-openshift-service-ca\") pod \"7c6f5108-546f-4a8f-93f4-960cf5838f69\" (UID: \"7c6f5108-546f-4a8f-93f4-960cf5838f69\") " Oct 30 16:32:56.289520 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.289493 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6f5108-546f-4a8f-93f4-960cf5838f69-openshift-service-ca" pod "7c6f5108-546f-4a8f-93f4-960cf5838f69" (UID: "7c6f5108-546f-4a8f-93f4-960cf5838f69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:32:56.291262 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.291229 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-maas-api-tls" pod "7c6f5108-546f-4a8f-93f4-960cf5838f69" (UID: "7c6f5108-546f-4a8f-93f4-960cf5838f69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:32:56.291372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.291303 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6f5108-546f-4a8f-93f4-960cf5838f69-kube-api-access-ww5zs" pod "7c6f5108-546f-4a8f-93f4-960cf5838f69" (UID: "7c6f5108-546f-4a8f-93f4-960cf5838f69"). InnerVolumeSpecName "kube-api-access-ww5zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:32:56.291372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.291308 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-metrics-certs" pod "7c6f5108-546f-4a8f-93f4-960cf5838f69" (UID: "7c6f5108-546f-4a8f-93f4-960cf5838f69"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:32:56.292964 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.292941 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:56.391492 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.391407 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:56.391492 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.391450 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6f5108-546f-4a8f-93f4-960cf5838f69-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:56.391492 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.391468 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww5zs\" (UniqueName: \"kubernetes.io/projected/7c6f5108-546f-4a8f-93f4-960cf5838f69-kube-api-access-ww5zs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:56.391492 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.391483 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6f5108-546f-4a8f-93f4-960cf5838f69-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:56.494772 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.494713 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:56.672981 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.672906 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:56.760769 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.760735 4861 generic.go:356] "Generic (PLEG): container finished" podID="7c6f5108-546f-4a8f-93f4-960cf5838f69" containerID="a38d0e4facb38d81486f4e023568e31ff5daa995546b99c7f59efc3924aedd46" exitCode=0 Oct 30 16:32:56.761234 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.760796 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" Oct 30 16:32:56.761234 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.760818 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" event={"ID":"7c6f5108-546f-4a8f-93f4-960cf5838f69","Type":"ContainerDied","Data":"a38d0e4facb38d81486f4e023568e31ff5daa995546b99c7f59efc3924aedd46"} Oct 30 16:32:56.761234 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.760896 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7" event={"ID":"7c6f5108-546f-4a8f-93f4-960cf5838f69","Type":"ContainerDied","Data":"f115d9566ff774cd5fd249d61dff19b18837eb36af7052faa4d1d2a3d0bda9aa"} Oct 30 16:32:56.761234 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.760912 4861 scope.go:122] "RemoveContainer" containerID="a38d0e4facb38d81486f4e023568e31ff5daa995546b99c7f59efc3924aedd46" Oct 30 16:32:56.770357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.770335 4861 scope.go:122] "RemoveContainer" containerID="a38d0e4facb38d81486f4e023568e31ff5daa995546b99c7f59efc3924aedd46" Oct 30 16:32:56.770607 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:32:56.770589 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38d0e4facb38d81486f4e023568e31ff5daa995546b99c7f59efc3924aedd46\": container with ID starting with a38d0e4facb38d81486f4e023568e31ff5daa995546b99c7f59efc3924aedd46 not found: ID does not exist" containerID="a38d0e4facb38d81486f4e023568e31ff5daa995546b99c7f59efc3924aedd46" Oct 30 16:32:56.770675 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.770615 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38d0e4facb38d81486f4e023568e31ff5daa995546b99c7f59efc3924aedd46"} err="failed to get container status \"a38d0e4facb38d81486f4e023568e31ff5daa995546b99c7f59efc3924aedd46\": rpc error: code = NotFound desc = could not find container \"a38d0e4facb38d81486f4e023568e31ff5daa995546b99c7f59efc3924aedd46\": container with ID starting with a38d0e4facb38d81486f4e023568e31ff5daa995546b99c7f59efc3924aedd46 not found: ID does not exist" Oct 30 16:32:56.852768 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.852745 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7"] Oct 30 16:32:56.856046 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.856025 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a2d47c16-5db884d7c-zcqg7"] Oct 30 16:32:56.871881 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:56.871858 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:57.191972 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.191934 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj"] Oct 30 16:32:57.192420 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.192402 4861 cpu_manager.go:414] "containerMap: removing container" podUID="22c3cd45-bf72-4b25-83d1-42a01952a1cf" containerName="payload-pre-processing" Oct 30 16:32:57.192512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.192421 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c3cd45-bf72-4b25-83d1-42a01952a1cf" containerName="payload-pre-processing" Oct 30 16:32:57.192512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.192437 4861 cpu_manager.go:414] "containerMap: removing container" podUID="edd32141-41dd-4b75-996e-f3c9fc03a396" containerName="maas-api" Oct 30 16:32:57.192512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.192444 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edd32141-41dd-4b75-996e-f3c9fc03a396" containerName="maas-api" Oct 30 16:32:57.192512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.192454 4861 cpu_manager.go:414] "containerMap: removing container" podUID="7c6f5108-546f-4a8f-93f4-960cf5838f69" containerName="maas-api" Oct 30 16:32:57.192512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.192460 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6f5108-546f-4a8f-93f4-960cf5838f69" containerName="maas-api" Oct 30 16:32:57.192512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.192488 4861 cpu_manager.go:414] "containerMap: removing container" podUID="d2197c23-4f38-406b-80fc-57daeb425e15" containerName="payload-processing" Oct 30 16:32:57.192512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.192494 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2197c23-4f38-406b-80fc-57daeb425e15" containerName="payload-processing" Oct 30 16:32:57.192512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.192514 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479" containerName="maas-api" Oct 30 16:32:57.192931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.192521 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479" containerName="maas-api" Oct 30 16:32:57.192931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.192616 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479" containerName="maas-api" Oct 30 16:32:57.192931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.192636 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c3cd45-bf72-4b25-83d1-42a01952a1cf" containerName="payload-pre-processing" Oct 30 16:32:57.192931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.192665 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="edd32141-41dd-4b75-996e-f3c9fc03a396" containerName="maas-api" Oct 30 16:32:57.192931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.192675 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6f5108-546f-4a8f-93f4-960cf5838f69" containerName="maas-api" Oct 30 16:32:57.192931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.192689 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2197c23-4f38-406b-80fc-57daeb425e15" containerName="payload-processing" Oct 30 16:32:57.197246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.197224 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.221661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.218217 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2d365c-openshift-default-dockercfg-gctmp\"" Oct 30 16:32:57.221803 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.221674 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj"] Oct 30 16:32:57.270912 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.270857 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:57.300226 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.300201 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.300358 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.300241 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.300358 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.300326 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.300476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.300361 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.300476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.300457 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.300567 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.300513 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.300622 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.300568 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.300622 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.300605 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.300759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.300626 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4lzj\" (UniqueName: \"kubernetes.io/projected/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-kube-api-access-c4lzj\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.347507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.347467 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck"] Oct 30 16:32:57.347838 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.347807 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" podUID="734244c1-0481-45e6-a479-5af996594d19" containerName="istio-proxy" containerID="cri-o://3f8bd6ced7c2dfd7e186818f69ff780298637df04cc677afcf199120a1cb8f90" gracePeriod=30 Oct 30 16:32:57.401468 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.401438 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.401602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.401483 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.401602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.401517 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.401602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.401547 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.401602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.401580 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.401602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.401595 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4lzj\" (UniqueName: \"kubernetes.io/projected/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-kube-api-access-c4lzj\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.401885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.401672 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.401885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.401691 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.401885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.401714 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.402041 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.401907 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.402041 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.401939 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.402182 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.402163 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.402243 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.402169 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.402876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.402847 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.404239 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.404213 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.404495 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.404476 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.424639 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.424611 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:57.425967 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.425950 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.426303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.426284 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4lzj\" (UniqueName: \"kubernetes.io/projected/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-kube-api-access-c4lzj\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.513952 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.513925 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:57.777795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.771182 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:57.807172 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.807135 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6f5108-546f-4a8f-93f4-960cf5838f69" path="/var/lib/kubelet/pods/7c6f5108-546f-4a8f-93f4-960cf5838f69/volumes" Oct 30 16:32:57.807787 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.807766 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479" path="/var/lib/kubelet/pods/b3bfc9f9-c2fd-443e-b4d3-2c5982b8f479/volumes" Oct 30 16:32:57.808193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.808177 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edd32141-41dd-4b75-996e-f3c9fc03a396" path="/var/lib/kubelet/pods/edd32141-41dd-4b75-996e-f3c9fc03a396/volumes" Oct 30 16:32:57.837615 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.837595 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj"] Oct 30 16:32:57.839393 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:32:57.839366 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12ea7c70_5553_4086_b4f5_dc5895c4f0a1.slice/crio-d5d5f5d6bb075edc869152c162cd9adb9d4d683b8bba0658e44610487f45807c WatchSource:0}: Error finding container d5d5f5d6bb075edc869152c162cd9adb9d4d683b8bba0658e44610487f45807c: Status 404 returned error can't find the container with id d5d5f5d6bb075edc869152c162cd9adb9d4d683b8bba0658e44610487f45807c Oct 30 16:32:57.841981 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.841952 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:57.842073 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.842017 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:57.842073 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:57.842060 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:58.014005 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.013974 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:58.182885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.182805 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:58.389084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.387909 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9"] Oct 30 16:32:58.389084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.388144 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" podUID="c0181dec-1f2a-4d48-b71a-5ce318ae042a" containerName="maas-api" containerID="cri-o://c612b617fb5fcae2fe11ce36de5b051208bf2ce535429af15d8c9d89bc482ecb" gracePeriod=30 Oct 30 16:32:58.482828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.482797 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:58.537421 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.537147 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7"] Oct 30 16:32:58.541349 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.541326 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.562199 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.562167 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f3503b-openshift-default-dockercfg-xgg9f\"" Oct 30 16:32:58.565758 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.565733 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7"] Oct 30 16:32:58.645763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.642037 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:58.665528 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.665500 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:58.716367 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.716335 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.716565 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.716385 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.716565 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.716402 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.716565 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.716513 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vrdt\" (UniqueName: \"kubernetes.io/projected/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-kube-api-access-9vrdt\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.716760 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.716580 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.716760 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.716625 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.716760 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.716706 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.716928 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.716809 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.716928 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.716859 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.794795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.794680 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" event={"ID":"12ea7c70-5553-4086-b4f5-dc5895c4f0a1","Type":"ContainerStarted","Data":"176aa82fc35ddbc16b790d118a27eb080fd0d796c4c2eb5045b36c9eb3b63a5f"} Oct 30 16:32:58.794795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.794771 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" event={"ID":"12ea7c70-5553-4086-b4f5-dc5895c4f0a1","Type":"ContainerStarted","Data":"d5d5f5d6bb075edc869152c162cd9adb9d4d683b8bba0658e44610487f45807c"} Oct 30 16:32:58.796096 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.796069 4861 generic.go:356] "Generic (PLEG): container finished" podID="c0181dec-1f2a-4d48-b71a-5ce318ae042a" containerID="c612b617fb5fcae2fe11ce36de5b051208bf2ce535429af15d8c9d89bc482ecb" exitCode=0 Oct 30 16:32:58.796226 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.796097 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" event={"ID":"c0181dec-1f2a-4d48-b71a-5ce318ae042a","Type":"ContainerDied","Data":"c612b617fb5fcae2fe11ce36de5b051208bf2ce535429af15d8c9d89bc482ecb"} Oct 30 16:32:58.796226 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.796155 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" event={"ID":"c0181dec-1f2a-4d48-b71a-5ce318ae042a","Type":"ContainerDied","Data":"8b7eb9811852bfeb93c9bdcac600378ec00e889375d77e03446bda63e341d24f"} Oct 30 16:32:58.796226 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.796112 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9" Oct 30 16:32:58.796226 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.796186 4861 scope.go:122] "RemoveContainer" containerID="c612b617fb5fcae2fe11ce36de5b051208bf2ce535429af15d8c9d89bc482ecb" Oct 30 16:32:58.807319 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.807301 4861 scope.go:122] "RemoveContainer" containerID="c612b617fb5fcae2fe11ce36de5b051208bf2ce535429af15d8c9d89bc482ecb" Oct 30 16:32:58.807609 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:32:58.807592 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c612b617fb5fcae2fe11ce36de5b051208bf2ce535429af15d8c9d89bc482ecb\": container with ID starting with c612b617fb5fcae2fe11ce36de5b051208bf2ce535429af15d8c9d89bc482ecb not found: ID does not exist" containerID="c612b617fb5fcae2fe11ce36de5b051208bf2ce535429af15d8c9d89bc482ecb" Oct 30 16:32:58.807663 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.807618 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c612b617fb5fcae2fe11ce36de5b051208bf2ce535429af15d8c9d89bc482ecb"} err="failed to get container status \"c612b617fb5fcae2fe11ce36de5b051208bf2ce535429af15d8c9d89bc482ecb\": rpc error: code = NotFound desc = could not find container \"c612b617fb5fcae2fe11ce36de5b051208bf2ce535429af15d8c9d89bc482ecb\": container with ID starting with c612b617fb5fcae2fe11ce36de5b051208bf2ce535429af15d8c9d89bc482ecb not found: ID does not exist" Oct 30 16:32:58.817217 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.817194 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0181dec-1f2a-4d48-b71a-5ce318ae042a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0181dec-1f2a-4d48-b71a-5ce318ae042a-maas-api-tls\") pod \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\" (UID: \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\") " Oct 30 16:32:58.817321 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.817292 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0181dec-1f2a-4d48-b71a-5ce318ae042a-kube-api-access-zvc7b\" (UniqueName: \"kubernetes.io/projected/c0181dec-1f2a-4d48-b71a-5ce318ae042a-kube-api-access-zvc7b\") pod \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\" (UID: \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\") " Oct 30 16:32:58.817379 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.817361 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0181dec-1f2a-4d48-b71a-5ce318ae042a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0181dec-1f2a-4d48-b71a-5ce318ae042a-metrics-certs\") pod \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\" (UID: \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\") " Oct 30 16:32:58.817437 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.817394 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0181dec-1f2a-4d48-b71a-5ce318ae042a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0181dec-1f2a-4d48-b71a-5ce318ae042a-openshift-service-ca\") pod \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\" (UID: \"c0181dec-1f2a-4d48-b71a-5ce318ae042a\") " Oct 30 16:32:58.817601 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.817578 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.817717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.817642 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.817717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.817678 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.817854 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.817718 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vrdt\" (UniqueName: \"kubernetes.io/projected/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-kube-api-access-9vrdt\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.817854 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.817796 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.817965 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.817846 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.817965 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.817890 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.818069 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.817962 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.818069 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.818005 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.818069 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.818062 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.818388 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.818359 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0181dec-1f2a-4d48-b71a-5ce318ae042a-openshift-service-ca" pod "c0181dec-1f2a-4d48-b71a-5ce318ae042a" (UID: "c0181dec-1f2a-4d48-b71a-5ce318ae042a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:32:58.818467 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.818379 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.818550 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.818528 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.818612 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.818584 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.818669 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.818621 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.819897 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.819875 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0181dec-1f2a-4d48-b71a-5ce318ae042a-metrics-certs" pod "c0181dec-1f2a-4d48-b71a-5ce318ae042a" (UID: "c0181dec-1f2a-4d48-b71a-5ce318ae042a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:32:58.820014 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.819978 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0181dec-1f2a-4d48-b71a-5ce318ae042a-maas-api-tls" pod "c0181dec-1f2a-4d48-b71a-5ce318ae042a" (UID: "c0181dec-1f2a-4d48-b71a-5ce318ae042a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:32:58.820507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.820489 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.820863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.820848 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0181dec-1f2a-4d48-b71a-5ce318ae042a-kube-api-access-zvc7b" pod "c0181dec-1f2a-4d48-b71a-5ce318ae042a" (UID: "c0181dec-1f2a-4d48-b71a-5ce318ae042a"). InnerVolumeSpecName "kube-api-access-zvc7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:32:58.821258 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.821243 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.850471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.850090 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vrdt\" (UniqueName: \"kubernetes.io/projected/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-kube-api-access-9vrdt\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.850807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.850783 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:58.850807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.850794 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.857432 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.857232 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:32:58.920081 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.919369 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0181dec-1f2a-4d48-b71a-5ce318ae042a-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:58.920081 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.919404 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvc7b\" (UniqueName: \"kubernetes.io/projected/c0181dec-1f2a-4d48-b71a-5ce318ae042a-kube-api-access-zvc7b\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:58.920081 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.919420 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0181dec-1f2a-4d48-b71a-5ce318ae042a-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:58.920081 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:58.919434 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0181dec-1f2a-4d48-b71a-5ce318ae042a-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:32:59.081425 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.081206 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:59.130991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.130952 4861 patch_prober.go:29] interesting pod/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:32:59.131159 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.131014 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" podUID="734244c1-0481-45e6-a479-5af996594d19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:32:59.141120 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.141075 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" podStartSLOduration=2.14106151 podStartE2EDuration="2.14106151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:58.845213879 +0000 UTC m=+609.187887917" watchObservedRunningTime="2026-10-30 16:32:59.14106151 +0000 UTC m=+609.483735548" Oct 30 16:32:59.143240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.143219 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9"] Oct 30 16:32:59.147492 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.147470 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cff59996-547cd54966-9ffs9"] Oct 30 16:32:59.203498 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.203472 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7"] Oct 30 16:32:59.204075 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:32:59.204054 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe89fc65_9620_4dc6_98da_cd1a3d57dbc3.slice/crio-fd0fe812b82c7b82e9c7ffd6be17dc223044105e95aabe6bcbd0ddcb3f33fc28 WatchSource:0}: Error finding container fd0fe812b82c7b82e9c7ffd6be17dc223044105e95aabe6bcbd0ddcb3f33fc28: Status 404 returned error can't find the container with id fd0fe812b82c7b82e9c7ffd6be17dc223044105e95aabe6bcbd0ddcb3f33fc28 Oct 30 16:32:59.207225 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.207185 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:59.207335 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.207314 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:59.207412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.207394 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:32:59.226169 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.226147 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:59.379966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.379890 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:32:59.514844 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.514815 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:59.519688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.519663 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:59.805686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.805651 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0181dec-1f2a-4d48-b71a-5ce318ae042a" path="/var/lib/kubelet/pods/c0181dec-1f2a-4d48-b71a-5ce318ae042a/volumes" Oct 30 16:32:59.806185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.806044 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:59.806185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.806086 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" event={"ID":"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3","Type":"ContainerStarted","Data":"74aadff8eee521b2089a67530089b6125daf98ab1faa898acd5a866e79aa0e73"} Oct 30 16:32:59.806185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.806157 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:32:59.806185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.806182 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" event={"ID":"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3","Type":"ContainerStarted","Data":"fd0fe812b82c7b82e9c7ffd6be17dc223044105e95aabe6bcbd0ddcb3f33fc28"} Oct 30 16:32:59.850991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.850946 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" podStartSLOduration=1.85092451 podStartE2EDuration="1.85092451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:59.849442718 +0000 UTC m=+610.192116757" watchObservedRunningTime="2026-10-30 16:32:59.85092451 +0000 UTC m=+610.193598547" Oct 30 16:32:59.857624 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:32:59.857605 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:33:00.890750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:00.890123 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:00.896759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:00.894762 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:33:01.186037 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.186015 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:33:01.347369 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.347336 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-workload-certs\") pod \"21b1c7fd-dba6-422b-bd28-92702748babc\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " Oct 30 16:33:01.347369 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.347373 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21b1c7fd-dba6-422b-bd28-92702748babc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b1c7fd-dba6-422b-bd28-92702748babc-istio-podinfo\") pod \"21b1c7fd-dba6-422b-bd28-92702748babc\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " Oct 30 16:33:01.347588 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.347428 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b1c7fd-dba6-422b-bd28-92702748babc-kube-api-access-9jcgj\" (UniqueName: \"kubernetes.io/projected/21b1c7fd-dba6-422b-bd28-92702748babc-kube-api-access-9jcgj\") pod \"21b1c7fd-dba6-422b-bd28-92702748babc\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " Oct 30 16:33:01.347588 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.347464 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-workload-socket\") pod \"21b1c7fd-dba6-422b-bd28-92702748babc\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " Oct 30 16:33:01.347691 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.347589 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b1c7fd-dba6-422b-bd28-92702748babc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b1c7fd-dba6-422b-bd28-92702748babc-istiod-ca-cert\") pod \"21b1c7fd-dba6-422b-bd28-92702748babc\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " Oct 30 16:33:01.347691 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.347646 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-workload-certs" pod "21b1c7fd-dba6-422b-bd28-92702748babc" (UID: "21b1c7fd-dba6-422b-bd28-92702748babc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:33:01.347843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.347702 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b1c7fd-dba6-422b-bd28-92702748babc-istio-token\" (UniqueName: \"kubernetes.io/projected/21b1c7fd-dba6-422b-bd28-92702748babc-istio-token\") pod \"21b1c7fd-dba6-422b-bd28-92702748babc\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " Oct 30 16:33:01.347843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.347751 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-workload-socket" pod "21b1c7fd-dba6-422b-bd28-92702748babc" (UID: "21b1c7fd-dba6-422b-bd28-92702748babc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:33:01.347843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.347780 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-credential-socket\") pod \"21b1c7fd-dba6-422b-bd28-92702748babc\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " Oct 30 16:33:01.347843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.347814 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-istio-envoy\") pod \"21b1c7fd-dba6-422b-bd28-92702748babc\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " Oct 30 16:33:01.348029 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.347887 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-istio-data\") pod \"21b1c7fd-dba6-422b-bd28-92702748babc\" (UID: \"21b1c7fd-dba6-422b-bd28-92702748babc\") " Oct 30 16:33:01.348079 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.348038 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-credential-socket" pod "21b1c7fd-dba6-422b-bd28-92702748babc" (UID: "21b1c7fd-dba6-422b-bd28-92702748babc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:33:01.348079 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.348060 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b1c7fd-dba6-422b-bd28-92702748babc-istiod-ca-cert" pod "21b1c7fd-dba6-422b-bd28-92702748babc" (UID: "21b1c7fd-dba6-422b-bd28-92702748babc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:33:01.348240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.348215 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-istio-data" pod "21b1c7fd-dba6-422b-bd28-92702748babc" (UID: "21b1c7fd-dba6-422b-bd28-92702748babc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:33:01.348502 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.348479 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:01.348502 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.348504 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b1c7fd-dba6-422b-bd28-92702748babc-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:01.348717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.348520 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:01.348717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.348534 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:01.348717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.348548 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:01.350035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.350009 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21b1c7fd-dba6-422b-bd28-92702748babc-istio-podinfo" pod "21b1c7fd-dba6-422b-bd28-92702748babc" (UID: "21b1c7fd-dba6-422b-bd28-92702748babc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:33:01.350035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.350024 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b1c7fd-dba6-422b-bd28-92702748babc-istio-token" pod "21b1c7fd-dba6-422b-bd28-92702748babc" (UID: "21b1c7fd-dba6-422b-bd28-92702748babc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:33:01.350203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.350169 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b1c7fd-dba6-422b-bd28-92702748babc-kube-api-access-9jcgj" pod "21b1c7fd-dba6-422b-bd28-92702748babc" (UID: "21b1c7fd-dba6-422b-bd28-92702748babc"). InnerVolumeSpecName "kube-api-access-9jcgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:33:01.350287 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.350202 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-istio-envoy" pod "21b1c7fd-dba6-422b-bd28-92702748babc" (UID: "21b1c7fd-dba6-422b-bd28-92702748babc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:33:01.379909 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.379885 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:01.449708 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.449617 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jcgj\" (UniqueName: \"kubernetes.io/projected/21b1c7fd-dba6-422b-bd28-92702748babc-kube-api-access-9jcgj\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:01.449708 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.449650 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b1c7fd-dba6-422b-bd28-92702748babc-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:01.449708 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.449661 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b1c7fd-dba6-422b-bd28-92702748babc-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:01.449708 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.449670 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b1c7fd-dba6-422b-bd28-92702748babc-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:01.540706 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.540677 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:01.687147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.687120 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:01.816444 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.816416 4861 generic.go:356] "Generic (PLEG): container finished" podID="21b1c7fd-dba6-422b-bd28-92702748babc" containerID="aa4a892bf499f16e577b511b0b9db869c2a5dfcbf03d751786af885bcb246195" exitCode=0 Oct 30 16:33:01.816578 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.816462 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" Oct 30 16:33:01.816578 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.816489 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" event={"ID":"21b1c7fd-dba6-422b-bd28-92702748babc","Type":"ContainerDied","Data":"aa4a892bf499f16e577b511b0b9db869c2a5dfcbf03d751786af885bcb246195"} Oct 30 16:33:01.816578 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.816545 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl" event={"ID":"21b1c7fd-dba6-422b-bd28-92702748babc","Type":"ContainerDied","Data":"f7175cf7bae8cf043b6039b32c74498c9e23dd339db8948604addb0a897b63f1"} Oct 30 16:33:01.816578 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.816571 4861 scope.go:122] "RemoveContainer" containerID="aa4a892bf499f16e577b511b0b9db869c2a5dfcbf03d751786af885bcb246195" Oct 30 16:33:01.817490 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.817086 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:33:01.818035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.817994 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:33:01.826702 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.826688 4861 scope.go:122] "RemoveContainer" containerID="d4ebf7ef297e8bed45e853d7bf78d498deb39e5f3451abdb0e78482a234bee85" Oct 30 16:33:01.835160 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.835141 4861 scope.go:122] "RemoveContainer" containerID="aa4a892bf499f16e577b511b0b9db869c2a5dfcbf03d751786af885bcb246195" Oct 30 16:33:01.835467 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:33:01.835446 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa4a892bf499f16e577b511b0b9db869c2a5dfcbf03d751786af885bcb246195\": container with ID starting with aa4a892bf499f16e577b511b0b9db869c2a5dfcbf03d751786af885bcb246195 not found: ID does not exist" containerID="aa4a892bf499f16e577b511b0b9db869c2a5dfcbf03d751786af885bcb246195" Oct 30 16:33:01.835543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.835476 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa4a892bf499f16e577b511b0b9db869c2a5dfcbf03d751786af885bcb246195"} err="failed to get container status \"aa4a892bf499f16e577b511b0b9db869c2a5dfcbf03d751786af885bcb246195\": rpc error: code = NotFound desc = could not find container \"aa4a892bf499f16e577b511b0b9db869c2a5dfcbf03d751786af885bcb246195\": container with ID starting with aa4a892bf499f16e577b511b0b9db869c2a5dfcbf03d751786af885bcb246195 not found: ID does not exist" Oct 30 16:33:01.835543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.835493 4861 scope.go:122] "RemoveContainer" containerID="d4ebf7ef297e8bed45e853d7bf78d498deb39e5f3451abdb0e78482a234bee85" Oct 30 16:33:01.835749 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:33:01.835714 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4ebf7ef297e8bed45e853d7bf78d498deb39e5f3451abdb0e78482a234bee85\": container with ID starting with d4ebf7ef297e8bed45e853d7bf78d498deb39e5f3451abdb0e78482a234bee85 not found: ID does not exist" containerID="d4ebf7ef297e8bed45e853d7bf78d498deb39e5f3451abdb0e78482a234bee85" Oct 30 16:33:01.835791 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.835751 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4ebf7ef297e8bed45e853d7bf78d498deb39e5f3451abdb0e78482a234bee85"} err="failed to get container status \"d4ebf7ef297e8bed45e853d7bf78d498deb39e5f3451abdb0e78482a234bee85\": rpc error: code = NotFound desc = could not find container \"d4ebf7ef297e8bed45e853d7bf78d498deb39e5f3451abdb0e78482a234bee85\": container with ID starting with d4ebf7ef297e8bed45e853d7bf78d498deb39e5f3451abdb0e78482a234bee85 not found: ID does not exist" Oct 30 16:33:01.882181 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.882158 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:01.939207 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.939176 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl"] Oct 30 16:33:01.941423 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:01.941404 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-37964af0-gw-openshift-default-778496cc8f-g4pkl"] Oct 30 16:33:02.050245 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.050215 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:02.214523 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.214440 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:02.379188 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.379162 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:02.528953 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.528915 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:02.618408 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.618385 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:33:02.764145 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764118 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-workload-certs\") pod \"734244c1-0481-45e6-a479-5af996594d19\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " Oct 30 16:33:02.764278 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764177 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-istio-data\") pod \"734244c1-0481-45e6-a479-5af996594d19\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " Oct 30 16:33:02.764278 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764207 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/734244c1-0481-45e6-a479-5af996594d19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734244c1-0481-45e6-a479-5af996594d19-istiod-ca-cert\") pod \"734244c1-0481-45e6-a479-5af996594d19\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " Oct 30 16:33:02.764278 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764233 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734244c1-0481-45e6-a479-5af996594d19-istio-token\" (UniqueName: \"kubernetes.io/projected/734244c1-0481-45e6-a479-5af996594d19-istio-token\") pod \"734244c1-0481-45e6-a479-5af996594d19\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " Oct 30 16:33:02.764278 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764263 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734244c1-0481-45e6-a479-5af996594d19-kube-api-access-t87fd\" (UniqueName: \"kubernetes.io/projected/734244c1-0481-45e6-a479-5af996594d19-kube-api-access-t87fd\") pod \"734244c1-0481-45e6-a479-5af996594d19\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " Oct 30 16:33:02.764469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764283 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-workload-socket\") pod \"734244c1-0481-45e6-a479-5af996594d19\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " Oct 30 16:33:02.764469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764377 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/734244c1-0481-45e6-a479-5af996594d19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734244c1-0481-45e6-a479-5af996594d19-istio-podinfo\") pod \"734244c1-0481-45e6-a479-5af996594d19\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " Oct 30 16:33:02.764469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764419 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-credential-socket\") pod \"734244c1-0481-45e6-a479-5af996594d19\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " Oct 30 16:33:02.764469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764416 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-workload-certs" pod "734244c1-0481-45e6-a479-5af996594d19" (UID: "734244c1-0481-45e6-a479-5af996594d19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:33:02.764469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764459 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-istio-envoy\") pod \"734244c1-0481-45e6-a479-5af996594d19\" (UID: \"734244c1-0481-45e6-a479-5af996594d19\") " Oct 30 16:33:02.764709 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764530 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-istio-data" pod "734244c1-0481-45e6-a479-5af996594d19" (UID: "734244c1-0481-45e6-a479-5af996594d19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:33:02.764813 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764616 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734244c1-0481-45e6-a479-5af996594d19-istiod-ca-cert" pod "734244c1-0481-45e6-a479-5af996594d19" (UID: "734244c1-0481-45e6-a479-5af996594d19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:33:02.764813 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764694 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-credential-socket" pod "734244c1-0481-45e6-a479-5af996594d19" (UID: "734244c1-0481-45e6-a479-5af996594d19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:33:02.764942 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764921 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:02.764988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764958 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734244c1-0481-45e6-a479-5af996594d19-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:02.764988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764973 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:02.765094 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764989 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:02.765094 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.764918 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-workload-socket" pod "734244c1-0481-45e6-a479-5af996594d19" (UID: "734244c1-0481-45e6-a479-5af996594d19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:33:02.766626 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.766598 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734244c1-0481-45e6-a479-5af996594d19-kube-api-access-t87fd" pod "734244c1-0481-45e6-a479-5af996594d19" (UID: "734244c1-0481-45e6-a479-5af996594d19"). InnerVolumeSpecName "kube-api-access-t87fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:33:02.766862 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.766834 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-istio-envoy" pod "734244c1-0481-45e6-a479-5af996594d19" (UID: "734244c1-0481-45e6-a479-5af996594d19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:33:02.767078 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.767061 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734244c1-0481-45e6-a479-5af996594d19-istio-token" pod "734244c1-0481-45e6-a479-5af996594d19" (UID: "734244c1-0481-45e6-a479-5af996594d19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:33:02.767161 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.767138 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/734244c1-0481-45e6-a479-5af996594d19-istio-podinfo" pod "734244c1-0481-45e6-a479-5af996594d19" (UID: "734244c1-0481-45e6-a479-5af996594d19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:33:02.824021 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.823990 4861 generic.go:356] "Generic (PLEG): container finished" podID="734244c1-0481-45e6-a479-5af996594d19" containerID="3f8bd6ced7c2dfd7e186818f69ff780298637df04cc677afcf199120a1cb8f90" exitCode=0 Oct 30 16:33:02.824148 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.824033 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" Oct 30 16:33:02.824148 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.824069 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" event={"ID":"734244c1-0481-45e6-a479-5af996594d19","Type":"ContainerDied","Data":"3f8bd6ced7c2dfd7e186818f69ff780298637df04cc677afcf199120a1cb8f90"} Oct 30 16:33:02.824148 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.824113 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck" event={"ID":"734244c1-0481-45e6-a479-5af996594d19","Type":"ContainerDied","Data":"3befaeb6b41cb6a2b77bf44cb92e231c1bf3afdb6d8cb9a1979a4aa0108cf588"} Oct 30 16:33:02.824148 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.824133 4861 scope.go:122] "RemoveContainer" containerID="3f8bd6ced7c2dfd7e186818f69ff780298637df04cc677afcf199120a1cb8f90" Oct 30 16:33:02.833856 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.833841 4861 scope.go:122] "RemoveContainer" containerID="e516133f5120d47a1d1795df515b2bcbe1873f7d73c68d8551582164eda05f99" Oct 30 16:33:02.842181 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.842164 4861 scope.go:122] "RemoveContainer" containerID="3f8bd6ced7c2dfd7e186818f69ff780298637df04cc677afcf199120a1cb8f90" Oct 30 16:33:02.842434 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:33:02.842415 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8bd6ced7c2dfd7e186818f69ff780298637df04cc677afcf199120a1cb8f90\": container with ID starting with 3f8bd6ced7c2dfd7e186818f69ff780298637df04cc677afcf199120a1cb8f90 not found: ID does not exist" containerID="3f8bd6ced7c2dfd7e186818f69ff780298637df04cc677afcf199120a1cb8f90" Oct 30 16:33:02.842475 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.842442 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8bd6ced7c2dfd7e186818f69ff780298637df04cc677afcf199120a1cb8f90"} err="failed to get container status \"3f8bd6ced7c2dfd7e186818f69ff780298637df04cc677afcf199120a1cb8f90\": rpc error: code = NotFound desc = could not find container \"3f8bd6ced7c2dfd7e186818f69ff780298637df04cc677afcf199120a1cb8f90\": container with ID starting with 3f8bd6ced7c2dfd7e186818f69ff780298637df04cc677afcf199120a1cb8f90 not found: ID does not exist" Oct 30 16:33:02.842475 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.842459 4861 scope.go:122] "RemoveContainer" containerID="e516133f5120d47a1d1795df515b2bcbe1873f7d73c68d8551582164eda05f99" Oct 30 16:33:02.842675 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:33:02.842661 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e516133f5120d47a1d1795df515b2bcbe1873f7d73c68d8551582164eda05f99\": container with ID starting with e516133f5120d47a1d1795df515b2bcbe1873f7d73c68d8551582164eda05f99 not found: ID does not exist" containerID="e516133f5120d47a1d1795df515b2bcbe1873f7d73c68d8551582164eda05f99" Oct 30 16:33:02.842712 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.842680 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e516133f5120d47a1d1795df515b2bcbe1873f7d73c68d8551582164eda05f99"} err="failed to get container status \"e516133f5120d47a1d1795df515b2bcbe1873f7d73c68d8551582164eda05f99\": rpc error: code = NotFound desc = could not find container \"e516133f5120d47a1d1795df515b2bcbe1873f7d73c68d8551582164eda05f99\": container with ID starting with e516133f5120d47a1d1795df515b2bcbe1873f7d73c68d8551582164eda05f99 not found: ID does not exist" Oct 30 16:33:02.866145 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.866122 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/734244c1-0481-45e6-a479-5af996594d19-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:02.866145 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.866144 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t87fd\" (UniqueName: \"kubernetes.io/projected/734244c1-0481-45e6-a479-5af996594d19-kube-api-access-t87fd\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:02.866261 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.866154 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:02.866261 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.866162 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734244c1-0481-45e6-a479-5af996594d19-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:02.866261 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.866170 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734244c1-0481-45e6-a479-5af996594d19-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:33:02.881350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.881237 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck"] Oct 30 16:33:02.883164 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:02.883146 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a2d47c16-openshift-default-6f7c9b4d97-7hdck"] Oct 30 16:33:03.098782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.098705 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:03.186988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.186957 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt"] Oct 30 16:33:03.187358 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.187347 4861 cpu_manager.go:414] "containerMap: removing container" podUID="21b1c7fd-dba6-422b-bd28-92702748babc" containerName="istio-proxy" Oct 30 16:33:03.187413 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.187360 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b1c7fd-dba6-422b-bd28-92702748babc" containerName="istio-proxy" Oct 30 16:33:03.187413 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.187373 4861 cpu_manager.go:414] "containerMap: removing container" podUID="734244c1-0481-45e6-a479-5af996594d19" containerName="istio-proxy" Oct 30 16:33:03.187413 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.187378 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734244c1-0481-45e6-a479-5af996594d19" containerName="istio-proxy" Oct 30 16:33:03.187413 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.187395 4861 cpu_manager.go:414] "containerMap: removing container" podUID="c0181dec-1f2a-4d48-b71a-5ce318ae042a" containerName="maas-api" Oct 30 16:33:03.187413 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.187400 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0181dec-1f2a-4d48-b71a-5ce318ae042a" containerName="maas-api" Oct 30 16:33:03.187598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.187459 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b1c7fd-dba6-422b-bd28-92702748babc" containerName="istio-proxy" Oct 30 16:33:03.187598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.187467 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="734244c1-0481-45e6-a479-5af996594d19" containerName="istio-proxy" Oct 30 16:33:03.187598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.187473 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b1c7fd-dba6-422b-bd28-92702748babc" containerName="istio-proxy" Oct 30 16:33:03.187598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.187479 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="734244c1-0481-45e6-a479-5af996594d19" containerName="istio-proxy" Oct 30 16:33:03.187598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.187486 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0181dec-1f2a-4d48-b71a-5ce318ae042a" containerName="maas-api" Oct 30 16:33:03.191921 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.191890 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" Oct 30 16:33:03.211059 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.211030 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-2d365c\"" Oct 30 16:33:03.211059 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.211053 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-2d365c-dockercfg-rz8ll\"" Oct 30 16:33:03.213335 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.213314 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt"] Oct 30 16:33:03.248461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.248434 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:03.258886 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.258842 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb"] Oct 30 16:33:03.259293 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.259278 4861 cpu_manager.go:414] "containerMap: removing container" podUID="21b1c7fd-dba6-422b-bd28-92702748babc" containerName="istio-proxy" Oct 30 16:33:03.259357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.259293 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b1c7fd-dba6-422b-bd28-92702748babc" containerName="istio-proxy" Oct 30 16:33:03.259357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.259306 4861 cpu_manager.go:414] "containerMap: removing container" podUID="734244c1-0481-45e6-a479-5af996594d19" containerName="istio-proxy" Oct 30 16:33:03.259357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.259311 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734244c1-0481-45e6-a479-5af996594d19" containerName="istio-proxy" Oct 30 16:33:03.262456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.262441 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" Oct 30 16:33:03.284089 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.284056 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb"] Oct 30 16:33:03.371271 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.371180 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgfl4\" (UniqueName: \"kubernetes.io/projected/04fa61ca-48d1-4fb0-99c9-2781775881d4-kube-api-access-vgfl4\") pod \"payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb\" (UID: \"04fa61ca-48d1-4fb0-99c9-2781775881d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" Oct 30 16:33:03.371271 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.371234 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m4zk\" (UniqueName: \"kubernetes.io/projected/6f5e2f42-7122-4574-b519-891d32498176-kube-api-access-9m4zk\") pod \"payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt\" (UID: \"6f5e2f42-7122-4574-b519-891d32498176\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" Oct 30 16:33:03.371485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.371342 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fa61ca-48d1-4fb0-99c9-2781775881d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb\" (UID: \"04fa61ca-48d1-4fb0-99c9-2781775881d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" Oct 30 16:33:03.371485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.371402 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f5e2f42-7122-4574-b519-891d32498176-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt\" (UID: \"6f5e2f42-7122-4574-b519-891d32498176\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" Oct 30 16:33:03.472775 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.472695 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fa61ca-48d1-4fb0-99c9-2781775881d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb\" (UID: \"04fa61ca-48d1-4fb0-99c9-2781775881d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" Oct 30 16:33:03.472957 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.472805 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f5e2f42-7122-4574-b519-891d32498176-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt\" (UID: \"6f5e2f42-7122-4574-b519-891d32498176\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" Oct 30 16:33:03.472957 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.472883 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgfl4\" (UniqueName: \"kubernetes.io/projected/04fa61ca-48d1-4fb0-99c9-2781775881d4-kube-api-access-vgfl4\") pod \"payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb\" (UID: \"04fa61ca-48d1-4fb0-99c9-2781775881d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" Oct 30 16:33:03.472957 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.472909 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m4zk\" (UniqueName: \"kubernetes.io/projected/6f5e2f42-7122-4574-b519-891d32498176-kube-api-access-9m4zk\") pod \"payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt\" (UID: \"6f5e2f42-7122-4574-b519-891d32498176\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" Oct 30 16:33:03.473556 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.473519 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fa61ca-48d1-4fb0-99c9-2781775881d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb\" (UID: \"04fa61ca-48d1-4fb0-99c9-2781775881d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" Oct 30 16:33:03.473679 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.473569 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f5e2f42-7122-4574-b519-891d32498176-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt\" (UID: \"6f5e2f42-7122-4574-b519-891d32498176\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" Oct 30 16:33:03.497305 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.497280 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m4zk\" (UniqueName: \"kubernetes.io/projected/6f5e2f42-7122-4574-b519-891d32498176-kube-api-access-9m4zk\") pod \"payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt\" (UID: \"6f5e2f42-7122-4574-b519-891d32498176\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" Oct 30 16:33:03.497444 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.497288 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgfl4\" (UniqueName: \"kubernetes.io/projected/04fa61ca-48d1-4fb0-99c9-2781775881d4-kube-api-access-vgfl4\") pod \"payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb\" (UID: \"04fa61ca-48d1-4fb0-99c9-2781775881d4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" Oct 30 16:33:03.505083 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.505051 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" Oct 30 16:33:03.567189 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.567159 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:03.574944 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.574825 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" Oct 30 16:33:03.806355 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.806324 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b1c7fd-dba6-422b-bd28-92702748babc" path="/var/lib/kubelet/pods/21b1c7fd-dba6-422b-bd28-92702748babc/volumes" Oct 30 16:33:03.806831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.806816 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734244c1-0481-45e6-a479-5af996594d19" path="/var/lib/kubelet/pods/734244c1-0481-45e6-a479-5af996594d19/volumes" Oct 30 16:33:03.833467 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.833448 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt"] Oct 30 16:33:03.835945 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:33:03.835922 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f5e2f42_7122_4574_b519_891d32498176.slice/crio-882fb668612f80c0c552b86ee4b398659a230a7820f2a7b9b80ebd53925338fd WatchSource:0}: Error finding container 882fb668612f80c0c552b86ee4b398659a230a7820f2a7b9b80ebd53925338fd: Status 404 returned error can't find the container with id 882fb668612f80c0c552b86ee4b398659a230a7820f2a7b9b80ebd53925338fd Oct 30 16:33:03.886955 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.886821 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb"] Oct 30 16:33:03.889836 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:33:03.889810 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04fa61ca_48d1_4fb0_99c9_2781775881d4.slice/crio-99427f1861fa597fb774538b6a012824056975facaab1f30784db2d6141d299e WatchSource:0}: Error finding container 99427f1861fa597fb774538b6a012824056975facaab1f30784db2d6141d299e: Status 404 returned error can't find the container with id 99427f1861fa597fb774538b6a012824056975facaab1f30784db2d6141d299e Oct 30 16:33:03.929780 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:03.929696 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:04.440632 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:04.440600 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:04.580013 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:04.579983 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:04.836452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:04.836421 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" event={"ID":"6f5e2f42-7122-4574-b519-891d32498176","Type":"ContainerStarted","Data":"adb282a7a2e1359c761b0a3012034b96b2958687ca4c0973967b20ae195b3177"} Oct 30 16:33:04.836625 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:04.836468 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" event={"ID":"6f5e2f42-7122-4574-b519-891d32498176","Type":"ContainerStarted","Data":"882fb668612f80c0c552b86ee4b398659a230a7820f2a7b9b80ebd53925338fd"} Oct 30 16:33:04.836625 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:04.836603 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" Oct 30 16:33:04.837963 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:04.837941 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" event={"ID":"04fa61ca-48d1-4fb0-99c9-2781775881d4","Type":"ContainerStarted","Data":"b924af75adf400410178ee6f8627da9d0c286db86955f5e6a92c926b0aaa615f"} Oct 30 16:33:04.838065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:04.837977 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" event={"ID":"04fa61ca-48d1-4fb0-99c9-2781775881d4","Type":"ContainerStarted","Data":"99427f1861fa597fb774538b6a012824056975facaab1f30784db2d6141d299e"} Oct 30 16:33:04.838065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:04.838012 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" Oct 30 16:33:04.880167 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:04.880117 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" podStartSLOduration=1.8801003189999999 podStartE2EDuration="1.880100319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:04.880011043 +0000 UTC m=+615.222685082" watchObservedRunningTime="2026-10-30 16:33:04.880100319 +0000 UTC m=+615.222774384" Oct 30 16:33:04.927558 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:04.927508 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" podStartSLOduration=1.92749456 podStartE2EDuration="1.92749456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:04.925506161 +0000 UTC m=+615.268180213" watchObservedRunningTime="2026-10-30 16:33:04.92749456 +0000 UTC m=+615.270168629" Oct 30 16:33:05.416305 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.416276 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2"] Oct 30 16:33:05.420372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.420345 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 16:33:05.435666 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.435641 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-2d365c-serving-cert\"" Oct 30 16:33:05.443263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.443219 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2"] Oct 30 16:33:05.551899 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.551860 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:05.595823 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.595794 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0891610-8a0f-41f6-937b-71b46b85defc-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2\" (UID: \"b0891610-8a0f-41f6-937b-71b46b85defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 16:33:05.595823 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.595824 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkbxw\" (UniqueName: \"kubernetes.io/projected/b0891610-8a0f-41f6-937b-71b46b85defc-kube-api-access-jkbxw\") pod \"maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2\" (UID: \"b0891610-8a0f-41f6-937b-71b46b85defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 16:33:05.596017 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.595869 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0891610-8a0f-41f6-937b-71b46b85defc-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2\" (UID: \"b0891610-8a0f-41f6-937b-71b46b85defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 16:33:05.596017 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.595967 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0891610-8a0f-41f6-937b-71b46b85defc-metrics-certs\") pod \"maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2\" (UID: \"b0891610-8a0f-41f6-937b-71b46b85defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 16:33:05.697361 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.697284 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0891610-8a0f-41f6-937b-71b46b85defc-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2\" (UID: \"b0891610-8a0f-41f6-937b-71b46b85defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 16:33:05.697361 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.697315 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkbxw\" (UniqueName: \"kubernetes.io/projected/b0891610-8a0f-41f6-937b-71b46b85defc-kube-api-access-jkbxw\") pod \"maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2\" (UID: \"b0891610-8a0f-41f6-937b-71b46b85defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 16:33:05.697581 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.697376 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0891610-8a0f-41f6-937b-71b46b85defc-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2\" (UID: \"b0891610-8a0f-41f6-937b-71b46b85defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 16:33:05.697581 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.697404 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0891610-8a0f-41f6-937b-71b46b85defc-metrics-certs\") pod \"maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2\" (UID: \"b0891610-8a0f-41f6-937b-71b46b85defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 16:33:05.698063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.698038 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0891610-8a0f-41f6-937b-71b46b85defc-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2\" (UID: \"b0891610-8a0f-41f6-937b-71b46b85defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 16:33:05.700031 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.700008 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0891610-8a0f-41f6-937b-71b46b85defc-maas-api-tls\") pod \"maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2\" (UID: \"b0891610-8a0f-41f6-937b-71b46b85defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 16:33:05.700133 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.700031 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0891610-8a0f-41f6-937b-71b46b85defc-metrics-certs\") pod \"maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2\" (UID: \"b0891610-8a0f-41f6-937b-71b46b85defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 16:33:05.701633 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.701612 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:05.712923 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.712902 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78"] Oct 30 16:33:05.717735 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.717702 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" Oct 30 16:33:05.727380 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.727349 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkbxw\" (UniqueName: \"kubernetes.io/projected/b0891610-8a0f-41f6-937b-71b46b85defc-kube-api-access-jkbxw\") pod \"maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2\" (UID: \"b0891610-8a0f-41f6-937b-71b46b85defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 16:33:05.734111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.734090 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 16:33:05.739499 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.739477 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78"] Oct 30 16:33:05.822074 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.822028 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-18519c\"" Oct 30 16:33:05.848546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.841074 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-18519c-dockercfg-l8nxz\"" Oct 30 16:33:05.867957 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.865652 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:05.899486 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.899446 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmtgg\" (UniqueName: \"kubernetes.io/projected/1c826c68-020f-4496-8c43-a78ba51f4e9c-kube-api-access-cmtgg\") pod \"payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78\" (UID: \"1c826c68-020f-4496-8c43-a78ba51f4e9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" Oct 30 16:33:05.899661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:05.899628 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c826c68-020f-4496-8c43-a78ba51f4e9c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78\" (UID: \"1c826c68-020f-4496-8c43-a78ba51f4e9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" Oct 30 16:33:06.000554 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.000515 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c826c68-020f-4496-8c43-a78ba51f4e9c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78\" (UID: \"1c826c68-020f-4496-8c43-a78ba51f4e9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" Oct 30 16:33:06.000767 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.000686 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmtgg\" (UniqueName: \"kubernetes.io/projected/1c826c68-020f-4496-8c43-a78ba51f4e9c-kube-api-access-cmtgg\") pod \"payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78\" (UID: \"1c826c68-020f-4496-8c43-a78ba51f4e9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" Oct 30 16:33:06.001339 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.001313 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c826c68-020f-4496-8c43-a78ba51f4e9c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78\" (UID: \"1c826c68-020f-4496-8c43-a78ba51f4e9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" Oct 30 16:33:06.029471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.029448 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmtgg\" (UniqueName: \"kubernetes.io/projected/1c826c68-020f-4496-8c43-a78ba51f4e9c-kube-api-access-cmtgg\") pod \"payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78\" (UID: \"1c826c68-020f-4496-8c43-a78ba51f4e9c\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" Oct 30 16:33:06.049648 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.049623 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:06.051578 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.051563 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" Oct 30 16:33:06.063017 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.062992 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2"] Oct 30 16:33:06.064612 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:33:06.064591 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0891610_8a0f_41f6_937b_71b46b85defc.slice/crio-b0b72c74c74f7ab5cc6cc3de7e9508d4bf82d4d3826ac486ede882023b701362 WatchSource:0}: Error finding container b0b72c74c74f7ab5cc6cc3de7e9508d4bf82d4d3826ac486ede882023b701362: Status 404 returned error can't find the container with id b0b72c74c74f7ab5cc6cc3de7e9508d4bf82d4d3826ac486ede882023b701362 Oct 30 16:33:06.207322 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.207291 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:06.356333 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.356307 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78"] Oct 30 16:33:06.358868 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:33:06.358845 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c826c68_020f_4496_8c43_a78ba51f4e9c.slice/crio-f776eb41693f60433146169aa18291e9a318836a16002e2ca9ef7dcaff058f83 WatchSource:0}: Error finding container f776eb41693f60433146169aa18291e9a318836a16002e2ca9ef7dcaff058f83: Status 404 returned error can't find the container with id f776eb41693f60433146169aa18291e9a318836a16002e2ca9ef7dcaff058f83 Oct 30 16:33:06.502080 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.502058 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:06.646316 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.646238 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:06.868018 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.867981 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" event={"ID":"1c826c68-020f-4496-8c43-a78ba51f4e9c","Type":"ContainerStarted","Data":"d97538a4feb421b23110b462a2e044000789b0a4d622a6e86619c930da2b2ed9"} Oct 30 16:33:06.868197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.868045 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" event={"ID":"1c826c68-020f-4496-8c43-a78ba51f4e9c","Type":"ContainerStarted","Data":"f776eb41693f60433146169aa18291e9a318836a16002e2ca9ef7dcaff058f83"} Oct 30 16:33:06.868197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.868075 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" Oct 30 16:33:06.869997 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.869962 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" event={"ID":"b0891610-8a0f-41f6-937b-71b46b85defc","Type":"ContainerStarted","Data":"6ff2ee59e8748d08e7cfd4370704ed6f81c4d276eef357bfae5b0ddf02c30ce9"} Oct 30 16:33:06.870151 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.870007 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" event={"ID":"b0891610-8a0f-41f6-937b-71b46b85defc","Type":"ContainerStarted","Data":"b0b72c74c74f7ab5cc6cc3de7e9508d4bf82d4d3826ac486ede882023b701362"} Oct 30 16:33:06.870151 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.870047 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 16:33:06.911096 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.911010 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" podStartSLOduration=1.9109961210000002 podStartE2EDuration="1.910996121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:06.910099326 +0000 UTC m=+617.252773361" watchObservedRunningTime="2026-10-30 16:33:06.910996121 +0000 UTC m=+617.253670158" Oct 30 16:33:06.954711 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:06.954668 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" podStartSLOduration=1.578907739 podStartE2EDuration="1.954656017s" totalImagesPullingTime="375.748278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:06.954239065 +0000 UTC m=+617.296913104" watchObservedRunningTime="2026-10-30 16:33:06.954656017 +0000 UTC m=+617.297330055" Oct 30 16:33:07.660835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:07.660804 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:07.821524 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:07.821497 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:07.983221 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:07.983191 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:08.952371 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:08.952340 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:09.032784 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.032755 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7"] Oct 30 16:33:09.036608 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.036586 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" Oct 30 16:33:09.067415 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.067389 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-420ff3\"" Oct 30 16:33:09.069449 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.069427 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7"] Oct 30 16:33:09.101146 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.101126 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:09.136303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.136273 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb294877-b8b7-45a5-b943-fc62473cedd7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7\" (UID: \"cb294877-b8b7-45a5-b943-fc62473cedd7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" Oct 30 16:33:09.136566 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.136544 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glrk7\" (UniqueName: \"kubernetes.io/projected/cb294877-b8b7-45a5-b943-fc62473cedd7-kube-api-access-glrk7\") pod \"payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7\" (UID: \"cb294877-b8b7-45a5-b943-fc62473cedd7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" Oct 30 16:33:09.237253 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.237213 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glrk7\" (UniqueName: \"kubernetes.io/projected/cb294877-b8b7-45a5-b943-fc62473cedd7-kube-api-access-glrk7\") pod \"payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7\" (UID: \"cb294877-b8b7-45a5-b943-fc62473cedd7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" Oct 30 16:33:09.237458 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.237272 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb294877-b8b7-45a5-b943-fc62473cedd7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7\" (UID: \"cb294877-b8b7-45a5-b943-fc62473cedd7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" Oct 30 16:33:09.238180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.238158 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb294877-b8b7-45a5-b943-fc62473cedd7-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7\" (UID: \"cb294877-b8b7-45a5-b943-fc62473cedd7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" Oct 30 16:33:09.261901 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.261868 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glrk7\" (UniqueName: \"kubernetes.io/projected/cb294877-b8b7-45a5-b943-fc62473cedd7-kube-api-access-glrk7\") pod \"payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7\" (UID: \"cb294877-b8b7-45a5-b943-fc62473cedd7\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" Oct 30 16:33:09.349973 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.349935 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" Oct 30 16:33:09.550063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.549404 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:09.673765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.666811 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7"] Oct 30 16:33:09.892446 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.892411 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" event={"ID":"cb294877-b8b7-45a5-b943-fc62473cedd7","Type":"ContainerStarted","Data":"3df7225512be5fca2a8f1f1a1fa033f26083e4c2759c8dce1708704cced338f3"} Oct 30 16:33:09.892629 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.892471 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" event={"ID":"cb294877-b8b7-45a5-b943-fc62473cedd7","Type":"ContainerStarted","Data":"1d28a3369333bd2c9889ea1cf93be0c67a623b304a7296a346f2f44271988d68"} Oct 30 16:33:09.892629 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.892500 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" Oct 30 16:33:09.946045 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:09.945988 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" podStartSLOduration=0.945972033 podStartE2EDuration="945.972033ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:09.943955584 +0000 UTC m=+620.286629621" watchObservedRunningTime="2026-10-30 16:33:09.945972033 +0000 UTC m=+620.288646073" Oct 30 16:33:10.085104 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:10.085008 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:10.233958 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:10.233925 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:10.428212 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:10.427944 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:10.575319 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:10.575289 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:11.233329 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:11.233291 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:11.695449 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:11.695366 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:11.864095 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:11.864062 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:12.545416 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:12.545382 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:12.880951 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:12.880876 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 16:33:14.663100 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:14.663036 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:14.828255 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:14.828196 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:14.965829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:14.965692 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:15.131831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:15.121701 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:15.864267 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:15.864238 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" Oct 30 16:33:15.864616 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:15.864303 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" Oct 30 16:33:16.761564 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:16.761529 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:16.912959 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:16.912875 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:17.074306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:17.074219 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:17.878965 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:17.878930 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" Oct 30 16:33:19.060011 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:19.059982 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:19.987954 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:19.987870 4861 generic.go:356] "Generic (PLEG): container finished" podID="12ea7c70-5553-4086-b4f5-dc5895c4f0a1" containerID="176aa82fc35ddbc16b790d118a27eb080fd0d796c4c2eb5045b36c9eb3b63a5f" exitCode=0 Oct 30 16:33:19.987954 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:19.987915 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" event={"ID":"12ea7c70-5553-4086-b4f5-dc5895c4f0a1","Type":"ContainerDied","Data":"176aa82fc35ddbc16b790d118a27eb080fd0d796c4c2eb5045b36c9eb3b63a5f"} Oct 30 16:33:19.988231 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:19.988207 4861 scope.go:122] "RemoveContainer" containerID="176aa82fc35ddbc16b790d118a27eb080fd0d796c4c2eb5045b36c9eb3b63a5f" Oct 30 16:33:19.989405 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:19.989379 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:33:19.989461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:19.989445 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:33:19.989496 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:19.989474 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:33:20.289300 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:33:20.289208 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:33:20.918747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:20.914915 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" Oct 30 16:33:20.997910 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:20.997878 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" event={"ID":"12ea7c70-5553-4086-b4f5-dc5895c4f0a1","Type":"ContainerStarted","Data":"2c63f603b2ce4f499af7a5b1ce96f86258fbe4902266e146b342284a67fc39b1"} Oct 30 16:33:21.238805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:21.238768 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:21.381803 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:21.381766 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:21.522589 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:21.522092 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:33:21.522589 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:21.522219 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:33:21.522589 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:21.522263 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" podUID="12ea7c70-5553-4086-b4f5-dc5895c4f0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 16:33:21.728772 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:21.728689 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:21.927450 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:21.927372 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:22.072544 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:22.072512 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:22.229447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:22.228346 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:22.373450 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:22.373403 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:22.526201 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:22.526108 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:33:22.526201 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:22.526170 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" podUID="12ea7c70-5553-4086-b4f5-dc5895c4f0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 16:33:22.527645 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:22.527614 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:23.495542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:23.495508 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:23.525747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:23.525082 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:33:23.525747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:23.525137 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" podUID="12ea7c70-5553-4086-b4f5-dc5895c4f0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 16:33:23.661012 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:23.660975 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:24.047865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:24.047778 4861 generic.go:356] "Generic (PLEG): container finished" podID="fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" containerID="74aadff8eee521b2089a67530089b6125daf98ab1faa898acd5a866e79aa0e73" exitCode=0 Oct 30 16:33:24.048050 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:24.047853 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" event={"ID":"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3","Type":"ContainerDied","Data":"74aadff8eee521b2089a67530089b6125daf98ab1faa898acd5a866e79aa0e73"} Oct 30 16:33:24.048310 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:24.048290 4861 scope.go:122] "RemoveContainer" containerID="74aadff8eee521b2089a67530089b6125daf98ab1faa898acd5a866e79aa0e73" Oct 30 16:33:24.049916 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:24.049879 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:33:24.050308 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:24.049972 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:33:24.050308 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:24.050016 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:33:24.529156 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:24.529117 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:33:24.529435 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:24.529405 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" podUID="12ea7c70-5553-4086-b4f5-dc5895c4f0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 16:33:25.060144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.060108 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" event={"ID":"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3","Type":"ContainerStarted","Data":"0ed9b3b072fac4b32f428cacde8c13d2eaa86031dbe367cb636569ef104fedcf"} Oct 30 16:33:25.497640 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.497597 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb"] Oct 30 16:33:25.515761 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.515012 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 16:33:25.519750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.518827 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Oct 30 16:33:25.519750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.518876 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" podUID="12ea7c70-5553-4086-b4f5-dc5895c4f0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Oct 30 16:33:25.525510 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.525478 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb"] Oct 30 16:33:25.531161 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.531134 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-a77b48-serving-cert\"" Oct 30 16:33:25.646380 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.646349 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:25.670438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.670412 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvmr8\" (UniqueName: \"kubernetes.io/projected/1d6544b2-2ae4-4901-a710-f37d04215115-kube-api-access-cvmr8\") pod \"maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb\" (UID: \"1d6544b2-2ae4-4901-a710-f37d04215115\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 16:33:25.670619 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.670607 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d6544b2-2ae4-4901-a710-f37d04215115-metrics-certs\") pod \"maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb\" (UID: \"1d6544b2-2ae4-4901-a710-f37d04215115\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 16:33:25.670784 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.670773 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d6544b2-2ae4-4901-a710-f37d04215115-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb\" (UID: \"1d6544b2-2ae4-4901-a710-f37d04215115\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 16:33:25.670900 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.670888 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6544b2-2ae4-4901-a710-f37d04215115-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb\" (UID: \"1d6544b2-2ae4-4901-a710-f37d04215115\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 16:33:25.772432 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.772358 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvmr8\" (UniqueName: \"kubernetes.io/projected/1d6544b2-2ae4-4901-a710-f37d04215115-kube-api-access-cvmr8\") pod \"maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb\" (UID: \"1d6544b2-2ae4-4901-a710-f37d04215115\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 16:33:25.772648 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.772629 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d6544b2-2ae4-4901-a710-f37d04215115-metrics-certs\") pod \"maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb\" (UID: \"1d6544b2-2ae4-4901-a710-f37d04215115\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 16:33:25.773565 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.772750 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d6544b2-2ae4-4901-a710-f37d04215115-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb\" (UID: \"1d6544b2-2ae4-4901-a710-f37d04215115\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 16:33:25.773565 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.772911 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6544b2-2ae4-4901-a710-f37d04215115-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb\" (UID: \"1d6544b2-2ae4-4901-a710-f37d04215115\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 16:33:25.773565 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.773526 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6544b2-2ae4-4901-a710-f37d04215115-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb\" (UID: \"1d6544b2-2ae4-4901-a710-f37d04215115\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 16:33:25.775284 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.775266 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d6544b2-2ae4-4901-a710-f37d04215115-metrics-certs\") pod \"maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb\" (UID: \"1d6544b2-2ae4-4901-a710-f37d04215115\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 16:33:25.775713 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.775697 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d6544b2-2ae4-4901-a710-f37d04215115-maas-api-tls\") pod \"maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb\" (UID: \"1d6544b2-2ae4-4901-a710-f37d04215115\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 16:33:25.792412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.792388 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:25.796381 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.796355 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvmr8\" (UniqueName: \"kubernetes.io/projected/1d6544b2-2ae4-4901-a710-f37d04215115-kube-api-access-cvmr8\") pod \"maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb\" (UID: \"1d6544b2-2ae4-4901-a710-f37d04215115\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 16:33:25.857771 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.857710 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:33:25.857940 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.857770 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:33:25.857940 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.857839 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" podUID="fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:33:25.864458 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:25.864437 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 16:33:26.232774 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:26.216702 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb"] Oct 30 16:33:26.237632 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:33:26.237601 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6544b2_2ae4_4901_a710_f37d04215115.slice/crio-a3fbed105c255c062388895346e305631c7ea1f9b99266c3a40992c79ce625d2 WatchSource:0}: Error finding container a3fbed105c255c062388895346e305631c7ea1f9b99266c3a40992c79ce625d2: Status 404 returned error can't find the container with id a3fbed105c255c062388895346e305631c7ea1f9b99266c3a40992c79ce625d2 Oct 30 16:33:26.535947 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:26.535855 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:33:26.536242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:26.536216 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:33:26.537012 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:26.536997 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:33:26.860797 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:26.860668 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:33:26.860797 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:26.860745 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" podUID="fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:33:27.073875 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:27.073836 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" event={"ID":"1d6544b2-2ae4-4901-a710-f37d04215115","Type":"ContainerStarted","Data":"ef674785f9a9edd66d853f135dcbd7fab1aa5e84a651b76b8903ed1d3cb93281"} Oct 30 16:33:27.074170 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:27.074153 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 16:33:27.074214 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:27.074197 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" event={"ID":"1d6544b2-2ae4-4901-a710-f37d04215115","Type":"ContainerStarted","Data":"a3fbed105c255c062388895346e305631c7ea1f9b99266c3a40992c79ce625d2"} Oct 30 16:33:27.444356 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:27.444322 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:27.713680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:27.713548 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:27.857961 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:27.857921 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:33:27.858139 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:27.857985 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" podUID="fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:33:27.978328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:27.978293 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:28.143976 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:28.143950 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:28.295591 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:28.295516 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:28.858477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:28.858440 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:33:28.858886 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:28.858497 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" podUID="fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:33:29.845001 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:29.844961 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:29.860410 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:29.860365 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 30 16:33:29.860922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:29.860426 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" podUID="fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 30 16:33:30.861845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:30.861814 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:33:30.862336 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:30.862218 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:33:30.862883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:30.862864 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 16:33:30.911994 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:30.911967 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:30.925161 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:30.925099 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" podStartSLOduration=5.568328692 podStartE2EDuration="5.925079463s" totalImagesPullingTime="356.750771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:33:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:33:27.122983201 +0000 UTC m=+637.465657239" watchObservedRunningTime="2026-10-30 16:33:30.925079463 +0000 UTC m=+641.267753505" Oct 30 16:33:31.483550 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:31.483519 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:31.626242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:31.626209 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:31.796605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:31.796523 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:31.947951 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:31.947916 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:33.085179 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:33.085152 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 16:33:33.892465 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:33.892430 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:35.691096 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:35.691058 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:37.494073 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:37.494041 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:39.309712 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:39.309673 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:40.106187 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:40.106142 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:40.479920 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:40.479888 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:40.684448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:40.684410 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:40.889785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:40.889698 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:41.034529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:41.034491 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:41.173881 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:41.173781 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:41.323345 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:41.323299 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:43.092904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:43.092875 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:43.237387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:43.237352 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:44.891772 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:44.891744 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:46.627781 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:46.627744 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:48.701176 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:48.701143 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:49.045137 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:49.045104 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:50.288798 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:33:50.288760 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:33:50.495488 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:50.495455 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:52.212021 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:52.211987 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:54.094067 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:54.094015 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:54.892587 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:54.892542 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:56.008949 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:56.008909 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:57.807366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:57.807330 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:33:59.611919 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:33:59.611883 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:01.390401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:01.390277 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:02.213973 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.213932 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn"] Oct 30 16:34:02.220704 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.220682 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.237704 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.237674 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn"] Oct 30 16:34:02.246771 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.246746 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"openshift-service-ca.crt\"" Oct 30 16:34:02.246771 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.246753 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"kube-root-ca.crt\"" Oct 30 16:34:02.248410 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.248391 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"premium-w2-18519c-kserve-self-signed-certs\"" Oct 30 16:34:02.248526 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.248496 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-18519c\"/\"default-dockercfg-rzlfm\"" Oct 30 16:34:02.291380 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.291353 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:02.379180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.379151 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-tls-certs\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.379295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.379206 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg9qt\" (UniqueName: \"kubernetes.io/projected/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-kube-api-access-tg9qt\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.379352 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.379333 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-model-cache\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.379389 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.379367 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-home\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.379423 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.379389 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-tmp-dir\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.379514 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.379496 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-dshm\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.481065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.481033 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg9qt\" (UniqueName: \"kubernetes.io/projected/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-kube-api-access-tg9qt\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.481463 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.481124 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-model-cache\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.481463 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.481149 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-home\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.481463 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.481166 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-tmp-dir\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.481463 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.481189 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-dshm\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.481463 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.481255 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-tls-certs\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.481769 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.481558 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-home\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.481769 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.481607 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-tmp-dir\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.481769 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.481636 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-model-cache\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.483622 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.483600 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-dshm\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.484040 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.484021 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-tls-certs\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.504593 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.504567 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg9qt\" (UniqueName: \"kubernetes.io/projected/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-kube-api-access-tg9qt\") pod \"premium-w2-18519c-kserve-58bf5cb4d5-fkgvn\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.534661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.534640 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:02.837153 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:02.837128 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn"] Oct 30 16:34:02.838867 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:34:02.838843 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf66112b3_864b_46bf_a8dd_3f22c04f8d3f.slice/crio-72ec8bf250709309f3320badba59b780edd04aa94c82e25de058839e702ea6b2 WatchSource:0}: Error finding container 72ec8bf250709309f3320badba59b780edd04aa94c82e25de058839e702ea6b2: Status 404 returned error can't find the container with id 72ec8bf250709309f3320badba59b780edd04aa94c82e25de058839e702ea6b2 Oct 30 16:34:03.115819 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:03.115118 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:03.265194 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:03.265155 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" event={"ID":"f66112b3-864b-46bf-a8dd-3f22c04f8d3f","Type":"ContainerStarted","Data":"73176821c7b674ca80e171e87d58797edb1a6486e06ba3b83755ccb5b3f0b686"} Oct 30 16:34:03.265369 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:03.265201 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" event={"ID":"f66112b3-864b-46bf-a8dd-3f22c04f8d3f","Type":"ContainerStarted","Data":"72ec8bf250709309f3320badba59b780edd04aa94c82e25de058839e702ea6b2"} Oct 30 16:34:03.310479 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:03.310433 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podStartSLOduration=1.310422525 podStartE2EDuration="1.310422525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:03.308683882 +0000 UTC m=+673.651357920" watchObservedRunningTime="2026-10-30 16:34:03.310422525 +0000 UTC m=+673.653096562" Oct 30 16:34:03.534967 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:03.534937 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:04.889408 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:04.889376 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:06.725077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:06.725044 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:06.874996 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:06.874965 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:08.507755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:08.507692 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:10.203103 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:10.203072 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:10.352495 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:10.352461 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:12.005675 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:12.005640 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:12.535656 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:12.535618 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:12.552164 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:12.552140 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:13.340635 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:13.340603 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:34:13.798692 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:13.798654 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:15.737745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:15.736859 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:17.594667 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:17.594641 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:18.256695 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:18.256663 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:19.415674 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:19.415642 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:20.288367 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:34:20.288334 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:34:21.184317 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:21.184273 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:22.995856 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:22.995821 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:24.852397 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:24.852338 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:26.565071 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:26.565041 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:26.715014 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:26.714916 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:28.585922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:28.585881 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:30.448495 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:30.448454 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:32.297558 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:32.297528 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:34.008258 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:34.008175 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:35.803685 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:35.803643 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:37.614846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:37.614816 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:39.415576 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:39.415545 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:41.212688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:41.212659 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:41.363248 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:41.363220 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:42.968966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:42.968929 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm"] Oct 30 16:34:42.973034 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:42.973009 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:42.990896 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:42.990876 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-18519c-openshift-default-dockercfg-nqftk\"" Oct 30 16:34:43.003896 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.003862 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm"] Oct 30 16:34:43.086063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.086033 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.086206 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.086068 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.086206 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.086088 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.086287 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.086225 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.086287 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.086250 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.086287 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.086268 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.086379 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.086289 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzpp5\" (UniqueName: \"kubernetes.io/projected/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-kube-api-access-vzpp5\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.086418 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.086407 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.086452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.086435 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.097466 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.097446 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:43.187799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.187768 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.187995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.187818 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.187995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.187850 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.187995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.187904 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.187995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.187929 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.187995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.187951 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.187995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.187984 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpp5\" (UniqueName: \"kubernetes.io/projected/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-kube-api-access-vzpp5\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.188329 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.188095 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.188329 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.188139 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.188329 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.188199 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.188506 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.188332 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.188506 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.188402 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.188506 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.188466 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.188830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.188810 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.190498 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.190478 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.190643 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.190624 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.211771 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.211713 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.211970 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.211955 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzpp5\" (UniqueName: \"kubernetes.io/projected/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-kube-api-access-vzpp5\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.286475 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.286401 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:43.632067 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.632030 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm"] Oct 30 16:34:43.639126 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.639071 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:34:43.639280 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.639226 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:34:43.639338 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:43.639312 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:34:44.535590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:44.535554 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" event={"ID":"5a3cf99a-1f00-4e00-891c-01b06d04a4b0","Type":"ContainerStarted","Data":"fa259f36cbf581a56c6ce685b896ee9956b8e4d17eee61ff66b307486f7bc744"} Oct 30 16:34:44.536022 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:44.535618 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" event={"ID":"5a3cf99a-1f00-4e00-891c-01b06d04a4b0","Type":"ContainerStarted","Data":"9ed7268cf18fc4ed98b6ebc2abf29602b3b2c655d24326912579d6d84d10a15d"} Oct 30 16:34:44.615400 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:44.615339 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" podStartSLOduration=2.6153196100000002 podStartE2EDuration="2.61531961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:44.594544196 +0000 UTC m=+714.937218235" watchObservedRunningTime="2026-10-30 16:34:44.61531961 +0000 UTC m=+714.957993649" Oct 30 16:34:45.033752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:45.033078 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:45.286878 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:45.286795 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:45.288166 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:45.288142 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:34:45.288340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:45.288187 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" podUID="5a3cf99a-1f00-4e00-891c-01b06d04a4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 16:34:46.286904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:46.286867 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:34:46.287368 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:46.286923 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" podUID="5a3cf99a-1f00-4e00-891c-01b06d04a4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 16:34:46.896529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:46.896496 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:47.287744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:47.287692 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:34:47.288129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:47.287773 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" podUID="5a3cf99a-1f00-4e00-891c-01b06d04a4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 16:34:48.286969 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:48.286937 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:34:48.287144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:48.286985 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" podUID="5a3cf99a-1f00-4e00-891c-01b06d04a4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 16:34:48.830131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:48.830085 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:49.287470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:49.287435 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:34:49.287640 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:49.287501 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" podUID="5a3cf99a-1f00-4e00-891c-01b06d04a4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Oct 30 16:34:50.291166 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:34:50.291108 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:34:50.311114 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:50.310711 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:50.311628 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:50.311595 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:50.312674 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:50.312656 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:34:50.737901 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:50.737870 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:51.180446 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:51.180363 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:51.321380 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:51.321352 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:52.528410 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:52.528375 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:54.286124 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:54.286091 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:56.057552 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:56.057518 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:56.936751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:56.936695 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj"] Oct 30 16:34:56.942447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:56.942423 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:56.964912 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:56.964873 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj"] Oct 30 16:34:56.974328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:56.974286 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a77b48-openshift-default-dockercfg-725dg\"" Oct 30 16:34:57.062466 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.062425 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021fd3dd-e215-430d-b702-b01f2e90053e-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.062466 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.062465 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.063084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.062493 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.063084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.062633 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.063084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.062666 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021fd3dd-e215-430d-b702-b01f2e90053e-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.063084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.062696 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.063084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.062715 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.063084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.062832 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfkrr\" (UniqueName: \"kubernetes.io/projected/021fd3dd-e215-430d-b702-b01f2e90053e-kube-api-access-qfkrr\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.063084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.062867 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/021fd3dd-e215-430d-b702-b01f2e90053e-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.164135 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.164104 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfkrr\" (UniqueName: \"kubernetes.io/projected/021fd3dd-e215-430d-b702-b01f2e90053e-kube-api-access-qfkrr\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.164135 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.164137 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/021fd3dd-e215-430d-b702-b01f2e90053e-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.164408 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.164154 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021fd3dd-e215-430d-b702-b01f2e90053e-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.164408 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.164181 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.164408 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.164223 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.164408 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.164313 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.164408 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.164363 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021fd3dd-e215-430d-b702-b01f2e90053e-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.164693 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.164412 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.164693 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.164441 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.164854 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.164820 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.164920 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.164898 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.164981 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.164968 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.165083 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.165062 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.165398 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.165375 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021fd3dd-e215-430d-b702-b01f2e90053e-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.167061 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.167038 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021fd3dd-e215-430d-b702-b01f2e90053e-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.167412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.167390 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.191194 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.191132 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfkrr\" (UniqueName: \"kubernetes.io/projected/021fd3dd-e215-430d-b702-b01f2e90053e-kube-api-access-qfkrr\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.193597 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.193571 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/021fd3dd-e215-430d-b702-b01f2e90053e-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.258167 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.257880 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:57.599802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.599772 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj"] Oct 30 16:34:57.602077 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:34:57.602031 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021fd3dd_e215_430d_b702_b01f2e90053e.slice/crio-4cc1398fe45554ace108a26ee975ce2f1af51a39cde59d3ac694894bfdf88069 WatchSource:0}: Error finding container 4cc1398fe45554ace108a26ee975ce2f1af51a39cde59d3ac694894bfdf88069: Status 404 returned error can't find the container with id 4cc1398fe45554ace108a26ee975ce2f1af51a39cde59d3ac694894bfdf88069 Oct 30 16:34:57.606012 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.605978 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:34:57.606137 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.606093 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:34:57.606338 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.606248 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:34:57.964316 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:57.964223 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:34:58.636804 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:58.636761 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" event={"ID":"021fd3dd-e215-430d-b702-b01f2e90053e","Type":"ContainerStarted","Data":"cffd064b2e612cdae4a41fd21e97c67057c190b3d0536dccc84b45046f0ba084"} Oct 30 16:34:58.637421 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:58.637397 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" event={"ID":"021fd3dd-e215-430d-b702-b01f2e90053e","Type":"ContainerStarted","Data":"4cc1398fe45554ace108a26ee975ce2f1af51a39cde59d3ac694894bfdf88069"} Oct 30 16:34:58.699836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:58.699785 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" podStartSLOduration=2.699770037 podStartE2EDuration="2.699770037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:58.698164348 +0000 UTC m=+729.040838399" watchObservedRunningTime="2026-10-30 16:34:58.699770037 +0000 UTC m=+729.042444076" Oct 30 16:34:59.258577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:59.258538 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:34:59.259981 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:59.259956 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:34:59.260078 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:59.260018 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" podUID="021fd3dd-e215-430d-b702-b01f2e90053e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 16:34:59.960617 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:34:59.960582 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:00.259687 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:00.259646 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:35:00.259905 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:00.259714 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" podUID="021fd3dd-e215-430d-b702-b01f2e90053e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 16:35:01.259702 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:01.259655 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:35:01.260130 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:01.259803 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" podUID="021fd3dd-e215-430d-b702-b01f2e90053e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 16:35:01.837787 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:01.837711 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:02.259763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:02.259706 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:35:02.260146 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:02.259788 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" podUID="021fd3dd-e215-430d-b702-b01f2e90053e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 16:35:03.259112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:03.259079 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:35:03.259295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:03.259132 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" podUID="021fd3dd-e215-430d-b702-b01f2e90053e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 30 16:35:03.783619 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:03.783569 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:04.409562 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:04.409531 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:35:04.409938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:04.409907 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:35:04.410734 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:04.410697 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:35:05.761621 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:05.761588 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:07.521138 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:07.521104 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:09.290506 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:09.290455 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:11.088346 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:11.080483 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:12.684175 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:12.684142 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:12.891489 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:12.891454 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:13.088129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.088091 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp"] Oct 30 16:35:13.092293 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.092274 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.121703 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.121674 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp"] Oct 30 16:35:13.169335 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.169301 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.169529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.169363 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.169529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.169469 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.169529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.169500 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.169684 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.169647 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.169746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.169733 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.169782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.169752 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zpfm\" (UniqueName: \"kubernetes.io/projected/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-kube-api-access-8zpfm\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.169881 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.169861 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.169918 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.169895 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.271455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.271424 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.271653 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.271462 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zpfm\" (UniqueName: \"kubernetes.io/projected/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-kube-api-access-8zpfm\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.271653 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.271493 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.271653 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.271516 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.271653 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.271581 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.271653 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.271622 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.271970 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.271667 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.271970 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.271695 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.271970 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.271812 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.272125 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.272009 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.272185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.272115 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.272185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.272145 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.272273 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.272217 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.272273 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.272247 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.274037 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.274016 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.274272 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.274253 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.295523 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.295497 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.295656 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.295636 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zpfm\" (UniqueName: \"kubernetes.io/projected/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-kube-api-access-8zpfm\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.406803 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.406707 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:13.727455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.727428 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp"] Oct 30 16:35:13.728625 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:35:13.728597 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866b3779_6bd0_4c61_a9cf_96161b1dc6d3.slice/crio-bce4b55aa9eb5f033b55f0ff3ba76487b943883e2cc6e21cfd127a5bf23326fe WatchSource:0}: Error finding container bce4b55aa9eb5f033b55f0ff3ba76487b943883e2cc6e21cfd127a5bf23326fe: Status 404 returned error can't find the container with id bce4b55aa9eb5f033b55f0ff3ba76487b943883e2cc6e21cfd127a5bf23326fe Oct 30 16:35:13.733003 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.732964 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:35:13.733124 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.733046 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:35:13.733124 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:13.733097 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:35:14.738643 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:14.738599 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" event={"ID":"866b3779-6bd0-4c61-a9cf-96161b1dc6d3","Type":"ContainerStarted","Data":"a8f67ecc31218e46f1f5a36386ee022d16cd83367e2dfb8406968ac3ab77bc39"} Oct 30 16:35:14.739059 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:14.738666 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" event={"ID":"866b3779-6bd0-4c61-a9cf-96161b1dc6d3","Type":"ContainerStarted","Data":"bce4b55aa9eb5f033b55f0ff3ba76487b943883e2cc6e21cfd127a5bf23326fe"} Oct 30 16:35:14.800364 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:14.800313 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" podStartSLOduration=1.800298111 podStartE2EDuration="1.800298111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:14.798167545 +0000 UTC m=+745.140841585" watchObservedRunningTime="2026-10-30 16:35:14.800298111 +0000 UTC m=+745.142972148" Oct 30 16:35:14.926372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:14.926284 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:15.127278 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:15.127194 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:15.415022 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:15.414938 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:15.415372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:15.415350 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:35:15.415535 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:15.415511 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" podUID="866b3779-6bd0-4c61-a9cf-96161b1dc6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 16:35:16.182166 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:16.182134 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:16.327189 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:16.327151 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:16.428640 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:16.428605 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:35:16.428974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:16.428945 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" podUID="866b3779-6bd0-4c61-a9cf-96161b1dc6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 16:35:16.491686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:16.491659 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:17.094044 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.094011 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:17.397180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.397105 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w"] Oct 30 16:35:17.406834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.404975 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:17.407469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.407343 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:35:17.408150 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.408123 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" podUID="866b3779-6bd0-4c61-a9cf-96161b1dc6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 16:35:17.446289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.446261 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w"] Oct 30 16:35:17.446545 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.446532 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 16:35:17.446762 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.446746 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 16:35:17.446858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.446832 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a169ff0f8c4c\"" Oct 30 16:35:17.446919 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.446877 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-83b62ed08788\"" Oct 30 16:35:17.447027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.447001 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-jb2w5\"" Oct 30 16:35:17.447104 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.447052 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-996a8bdbc296\"" Oct 30 16:35:17.447104 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.447097 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 16:35:17.447589 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.447567 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-503ee2727cb9\"" Oct 30 16:35:17.447701 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.447641 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 16:35:17.459891 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.459869 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq"] Oct 30 16:35:17.465003 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.464983 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.474145 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.474123 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq"] Oct 30 16:35:17.488060 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.488030 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.488179 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.488100 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7406f81-87cd-42a1-b4bd-237006ec5058-provider-credentials\") pod \"payload-processing-external-model-795bdd7876-qxl8w\" (UID: \"f7406f81-87cd-42a1-b4bd-237006ec5058\") " pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:17.488239 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.488194 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7406f81-87cd-42a1-b4bd-237006ec5058-plugins-config-volume\") pod \"payload-processing-external-model-795bdd7876-qxl8w\" (UID: \"f7406f81-87cd-42a1-b4bd-237006ec5058\") " pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:17.488239 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.488233 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.488352 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.488264 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mqwg\" (UniqueName: \"kubernetes.io/projected/f7406f81-87cd-42a1-b4bd-237006ec5058-kube-api-access-8mqwg\") pod \"payload-processing-external-model-795bdd7876-qxl8w\" (UID: \"f7406f81-87cd-42a1-b4bd-237006ec5058\") " pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:17.488352 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.488290 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2973e4-f250-465b-af94-f3fd93960586-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.488352 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.488326 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.488510 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.488366 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2973e4-f250-465b-af94-f3fd93960586-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.488510 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.488397 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mxxm\" (UniqueName: \"kubernetes.io/projected/dd2973e4-f250-465b-af94-f3fd93960586-kube-api-access-8mxxm\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.488510 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.488435 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.488656 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.488499 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2973e4-f250-465b-af94-f3fd93960586-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.488656 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.488560 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.488656 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.488630 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7406f81-87cd-42a1-b4bd-237006ec5058-routing-overlay-volume\") pod \"payload-processing-external-model-795bdd7876-qxl8w\" (UID: \"f7406f81-87cd-42a1-b4bd-237006ec5058\") " pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:17.537749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.529075 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-420ff3-openshift-default-dockercfg-mlfh4\"" Oct 30 16:35:17.589934 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.589898 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.590111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.589964 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7406f81-87cd-42a1-b4bd-237006ec5058-provider-credentials\") pod \"payload-processing-external-model-795bdd7876-qxl8w\" (UID: \"f7406f81-87cd-42a1-b4bd-237006ec5058\") " pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:17.590111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.590003 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7406f81-87cd-42a1-b4bd-237006ec5058-plugins-config-volume\") pod \"payload-processing-external-model-795bdd7876-qxl8w\" (UID: \"f7406f81-87cd-42a1-b4bd-237006ec5058\") " pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:17.590111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.590033 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.590111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.590062 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mqwg\" (UniqueName: \"kubernetes.io/projected/f7406f81-87cd-42a1-b4bd-237006ec5058-kube-api-access-8mqwg\") pod \"payload-processing-external-model-795bdd7876-qxl8w\" (UID: \"f7406f81-87cd-42a1-b4bd-237006ec5058\") " pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:17.590111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.590088 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2973e4-f250-465b-af94-f3fd93960586-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.590393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.590122 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.590393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.590160 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2973e4-f250-465b-af94-f3fd93960586-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.590393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.590188 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mxxm\" (UniqueName: \"kubernetes.io/projected/dd2973e4-f250-465b-af94-f3fd93960586-kube-api-access-8mxxm\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.590393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.590227 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.590393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.590286 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2973e4-f250-465b-af94-f3fd93960586-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.590393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.590326 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.590393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.590351 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.590393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.590393 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7406f81-87cd-42a1-b4bd-237006ec5058-routing-overlay-volume\") pod \"payload-processing-external-model-795bdd7876-qxl8w\" (UID: \"f7406f81-87cd-42a1-b4bd-237006ec5058\") " pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:17.590838 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.590608 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.591161 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.591145 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.591251 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.591224 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.592029 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.591800 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2973e4-f250-465b-af94-f3fd93960586-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.592140 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.592077 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7406f81-87cd-42a1-b4bd-237006ec5058-plugins-config-volume\") pod \"payload-processing-external-model-795bdd7876-qxl8w\" (UID: \"f7406f81-87cd-42a1-b4bd-237006ec5058\") " pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:17.592319 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.592291 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7406f81-87cd-42a1-b4bd-237006ec5058-routing-overlay-volume\") pod \"payload-processing-external-model-795bdd7876-qxl8w\" (UID: \"f7406f81-87cd-42a1-b4bd-237006ec5058\") " pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:17.593829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.593794 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2973e4-f250-465b-af94-f3fd93960586-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.594547 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.594521 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.595258 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.595235 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7406f81-87cd-42a1-b4bd-237006ec5058-provider-credentials\") pod \"payload-processing-external-model-795bdd7876-qxl8w\" (UID: \"f7406f81-87cd-42a1-b4bd-237006ec5058\") " pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:17.625744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.618738 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mqwg\" (UniqueName: \"kubernetes.io/projected/f7406f81-87cd-42a1-b4bd-237006ec5058-kube-api-access-8mqwg\") pod \"payload-processing-external-model-795bdd7876-qxl8w\" (UID: \"f7406f81-87cd-42a1-b4bd-237006ec5058\") " pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:17.627074 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.626348 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2973e4-f250-465b-af94-f3fd93960586-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.627074 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.627024 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mxxm\" (UniqueName: \"kubernetes.io/projected/dd2973e4-f250-465b-af94-f3fd93960586-kube-api-access-8mxxm\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:17.739013 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.738760 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:17.786937 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:17.786902 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:18.092342 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:35:18.092312 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7406f81_87cd_42a1_b4bd_237006ec5058.slice/crio-8fb8480b201833b85acc8c5a08e853c79b21b264a96170dd689905c66c54c809 WatchSource:0}: Error finding container 8fb8480b201833b85acc8c5a08e853c79b21b264a96170dd689905c66c54c809: Status 404 returned error can't find the container with id 8fb8480b201833b85acc8c5a08e853c79b21b264a96170dd689905c66c54c809 Oct 30 16:35:18.092715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:18.092686 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w"] Oct 30 16:35:18.250848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:18.250810 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq"] Oct 30 16:35:18.272749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:18.272091 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:35:18.272749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:18.272180 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:35:18.272749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:18.272223 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:35:18.416790 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:18.410089 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:35:18.416790 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:18.410142 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" podUID="866b3779-6bd0-4c61-a9cf-96161b1dc6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 16:35:18.797754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:18.796949 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" event={"ID":"dd2973e4-f250-465b-af94-f3fd93960586","Type":"ContainerStarted","Data":"76620f23a2f688708c2ce10d93e1a98e961a99b20358bdbf0ee79ef7acd1017b"} Oct 30 16:35:18.797754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:18.797019 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" event={"ID":"dd2973e4-f250-465b-af94-f3fd93960586","Type":"ContainerStarted","Data":"848786dc07de0661b385ebc82a278607dac33db67ea08b70aa1180566eccb1f0"} Oct 30 16:35:18.799321 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:18.799291 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" event={"ID":"f7406f81-87cd-42a1-b4bd-237006ec5058","Type":"ContainerStarted","Data":"20bc5d701cdb3f7c70fe78452e5d08a9aac3624ea048227b7db7a407df1007b3"} Oct 30 16:35:18.799532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:18.799512 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" event={"ID":"f7406f81-87cd-42a1-b4bd-237006ec5058","Type":"ContainerStarted","Data":"8fb8480b201833b85acc8c5a08e853c79b21b264a96170dd689905c66c54c809"} Oct 30 16:35:18.799857 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:18.799830 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:18.914147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:18.914077 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" podStartSLOduration=1.91405761 podStartE2EDuration="1.91405761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:18.913185355 +0000 UTC m=+749.255859395" watchObservedRunningTime="2026-10-30 16:35:18.91405761 +0000 UTC m=+749.256731643" Oct 30 16:35:18.917389 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:18.917343 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" podStartSLOduration=1.91732999 podStartE2EDuration="1.91732999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:18.86503614 +0000 UTC m=+749.207710179" watchObservedRunningTime="2026-10-30 16:35:18.91732999 +0000 UTC m=+749.260004029" Oct 30 16:35:19.413120 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:19.413081 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:35:19.413814 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:19.413783 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" podUID="866b3779-6bd0-4c61-a9cf-96161b1dc6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 16:35:19.419736 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:19.419691 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:19.788844 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:19.788807 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:19.790349 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:19.790319 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:35:19.790489 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:19.790375 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" podUID="dd2973e4-f250-465b-af94-f3fd93960586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 16:35:20.289286 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:35:20.289247 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:35:20.713165 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:20.713083 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:20.715111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:20.715089 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:20.725022 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:20.725002 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:35:20.787410 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:20.787383 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:35:20.787548 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:20.787436 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" podUID="dd2973e4-f250-465b-af94-f3fd93960586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 16:35:21.562130 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:21.562100 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:21.790143 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:21.790114 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:35:21.790700 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:21.790671 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" podUID="dd2973e4-f250-465b-af94-f3fd93960586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 16:35:22.788317 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:22.788255 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:35:22.788317 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:22.788306 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" podUID="dd2973e4-f250-465b-af94-f3fd93960586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 16:35:23.597033 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:23.596997 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:23.788077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:23.788045 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:35:23.788304 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:23.788275 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" podUID="dd2973e4-f250-465b-af94-f3fd93960586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 16:35:23.799970 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:23.799922 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:24.690864 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:24.690835 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-cfqjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:35:24.691347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:24.691324 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" podUID="c2cb80d7-4995-4584-8781-0c5921d6115c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:35:24.788234 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:24.788200 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:35:24.788542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:24.788499 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" podUID="dd2973e4-f250-465b-af94-f3fd93960586" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Oct 30 16:35:25.723745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:25.715105 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:26.512507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:26.512475 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:26.513486 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:26.513455 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:26.514966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:26.514943 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:35:27.583545 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:27.583511 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:29.388516 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:29.388483 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:29.872964 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:29.872938 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:31.198149 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:31.198052 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:33.044750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:33.044435 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:34.882553 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:34.882518 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:35.058418 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:35.058373 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:36.623173 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:36.623135 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:37.040057 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:37.040019 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:38.907996 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:38.907879 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:40.613610 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:40.613540 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:41.324787 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:41.324716 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:42.043174 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.043144 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592"] Oct 30 16:35:42.047128 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.047107 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.090770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.087133 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592"] Oct 30 16:35:42.208555 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.208524 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.208717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.208575 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5qzc\" (UniqueName: \"kubernetes.io/projected/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-kube-api-access-g5qzc\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.208717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.208604 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.208717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.208664 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.208717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.208694 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.208930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.208813 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.208930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.208854 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.209029 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.208952 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.209086 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.209039 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.310617 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.310536 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.310617 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.310609 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.310873 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.310640 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.310873 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.310674 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.310873 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.310715 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.310873 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.310817 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.311062 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.310879 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.311062 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.310947 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.311062 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.311008 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5qzc\" (UniqueName: \"kubernetes.io/projected/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-kube-api-access-g5qzc\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.311186 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.311137 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.311227 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.311190 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.311497 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.311466 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.311782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.311479 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.311782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.311503 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.313325 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.313291 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.313416 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.313390 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.338827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.338800 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5qzc\" (UniqueName: \"kubernetes.io/projected/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-kube-api-access-g5qzc\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.342184 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.342158 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-x7592\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.361218 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.361198 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:42.441870 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.431262 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:42.751075 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.751049 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592"] Oct 30 16:35:42.751954 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:35:42.751930 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e22d1e7_4404_470d_9bda_3fd5e1a67ed7.slice/crio-7050ec73eb47fef325b8e70a02d68edbaa6aaaef918848c2999ddffbef7915f7 WatchSource:0}: Error finding container 7050ec73eb47fef325b8e70a02d68edbaa6aaaef918848c2999ddffbef7915f7: Status 404 returned error can't find the container with id 7050ec73eb47fef325b8e70a02d68edbaa6aaaef918848c2999ddffbef7915f7 Oct 30 16:35:42.754158 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.754123 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:35:42.754259 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.754200 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:35:42.754259 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.754243 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:35:42.983291 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.983249 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" event={"ID":"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7","Type":"ContainerStarted","Data":"f08942ba360fbfa68e374e3eb5c9834484277675a096f25fa24421deb31012db"} Oct 30 16:35:42.983449 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:42.983314 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" event={"ID":"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7","Type":"ContainerStarted","Data":"7050ec73eb47fef325b8e70a02d68edbaa6aaaef918848c2999ddffbef7915f7"} Oct 30 16:35:43.362466 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:43.362436 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:43.363703 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:43.363674 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x7592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:35:43.363846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:43.363745 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" podUID="5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 16:35:44.366153 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:44.366067 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x7592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:35:44.366778 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:44.366708 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" podUID="5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 16:35:44.694955 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:44.694818 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:45.363106 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:45.363070 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x7592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:35:45.363390 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:45.363360 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" podUID="5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 16:35:46.362908 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:46.361787 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x7592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:35:46.362908 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:46.361861 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" podUID="5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 16:35:46.725596 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:46.725565 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:47.361978 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:47.361945 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x7592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:35:47.362142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:47.361989 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" podUID="5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 16:35:48.362606 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:48.362570 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x7592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:35:48.363114 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:48.362633 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" podUID="5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 16:35:48.721637 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:48.721604 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:49.356995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:49.356933 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" podStartSLOduration=7.356913635 podStartE2EDuration="7.356913635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:43.058759263 +0000 UTC m=+773.401433300" watchObservedRunningTime="2026-10-30 16:35:49.356913635 +0000 UTC m=+779.699587674" Oct 30 16:35:49.359442 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:49.359416 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w"] Oct 30 16:35:49.361596 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:49.361574 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x7592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:35:49.361666 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:49.361623 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" podUID="5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 16:35:50.289888 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:35:50.289856 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:35:50.361571 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:50.361534 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x7592 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:35:50.361783 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:50.361603 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" podUID="5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 16:35:50.513245 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:50.513188 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:50.813012 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:50.812955 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:50.966756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:50.965751 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:51.180053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.179957 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh"] Oct 30 16:35:51.186069 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.186046 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:35:51.205807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.205778 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh"] Oct 30 16:35:51.295040 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.295011 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-routing-overlay-volume\") pod \"payload-processing-external-model-7cb69f8d98-mb9vh\" (UID: \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\") " pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:35:51.295465 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.295085 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-plugins-config-volume\") pod \"payload-processing-external-model-7cb69f8d98-mb9vh\" (UID: \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\") " pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:35:51.295465 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.295208 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials\") pod \"payload-processing-external-model-7cb69f8d98-mb9vh\" (UID: \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\") " pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:35:51.295465 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.295366 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jpd6\" (UniqueName: \"kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-kube-api-access-8jpd6\") pod \"payload-processing-external-model-7cb69f8d98-mb9vh\" (UID: \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\") " pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:35:51.396867 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.396835 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-plugins-config-volume\") pod \"payload-processing-external-model-7cb69f8d98-mb9vh\" (UID: \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\") " pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:35:51.397035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.396880 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials\") pod \"payload-processing-external-model-7cb69f8d98-mb9vh\" (UID: \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\") " pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:35:51.397035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.396951 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jpd6\" (UniqueName: \"kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-kube-api-access-8jpd6\") pod \"payload-processing-external-model-7cb69f8d98-mb9vh\" (UID: \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\") " pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:35:51.397035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.397004 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-routing-overlay-volume\") pod \"payload-processing-external-model-7cb69f8d98-mb9vh\" (UID: \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\") " pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:35:51.397628 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.397593 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-routing-overlay-volume\") pod \"payload-processing-external-model-7cb69f8d98-mb9vh\" (UID: \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\") " pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:35:51.397805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.397740 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-plugins-config-volume\") pod \"payload-processing-external-model-7cb69f8d98-mb9vh\" (UID: \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\") " pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:35:51.399715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.399690 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials\") pod \"payload-processing-external-model-7cb69f8d98-mb9vh\" (UID: \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\") " pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:35:51.421464 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.421438 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jpd6\" (UniqueName: \"kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-kube-api-access-8jpd6\") pod \"payload-processing-external-model-7cb69f8d98-mb9vh\" (UID: \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\") " pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:35:51.525798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.519960 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:35:51.889207 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.889116 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:51.890049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.889662 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:51.891169 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.891146 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:35:51.910677 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:35:51.910644 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225f7152_a7d6_4b53_b705_ad3ea3abf1c9.slice/crio-1fd45b779356d54d132bed9b9f2065247010f7b54f6ef0e4b5886437c0425e79 WatchSource:0}: Error finding container 1fd45b779356d54d132bed9b9f2065247010f7b54f6ef0e4b5886437c0425e79: Status 404 returned error can't find the container with id 1fd45b779356d54d132bed9b9f2065247010f7b54f6ef0e4b5886437c0425e79 Oct 30 16:35:51.911114 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:51.911092 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh"] Oct 30 16:35:52.118979 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:52.118920 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" event={"ID":"225f7152-a7d6-4b53-b705-ad3ea3abf1c9","Type":"ContainerStarted","Data":"aa756f127147fef698470df35eb6ead5713bea2840ef8374c77a2badf4c885e5"} Oct 30 16:35:52.118979 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:52.118988 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" event={"ID":"225f7152-a7d6-4b53-b705-ad3ea3abf1c9","Type":"ContainerStarted","Data":"1fd45b779356d54d132bed9b9f2065247010f7b54f6ef0e4b5886437c0425e79"} Oct 30 16:35:52.119249 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:52.119019 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:35:52.520799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:52.520374 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:54.360343 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:54.360302 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" podUID="f7406f81-87cd-42a1-b4bd-237006ec5058" containerName="payload-processing" containerID="cri-o://20bc5d701cdb3f7c70fe78452e5d08a9aac3624ea048227b7db7a407df1007b3" gracePeriod=25 Oct 30 16:35:54.390134 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:54.390106 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:54.706642 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:54.701111 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:54.746582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:54.746475 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" podStartSLOduration=3.746457915 podStartE2EDuration="3.746457915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:52.184955493 +0000 UTC m=+782.527629532" watchObservedRunningTime="2026-10-30 16:35:54.746457915 +0000 UTC m=+785.089131955" Oct 30 16:35:54.761184 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:54.761151 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7406f81-87cd-42a1-b4bd-237006ec5058-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7406f81-87cd-42a1-b4bd-237006ec5058-provider-credentials\") pod \"f7406f81-87cd-42a1-b4bd-237006ec5058\" (UID: \"f7406f81-87cd-42a1-b4bd-237006ec5058\") " Oct 30 16:35:54.761354 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:54.761228 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7406f81-87cd-42a1-b4bd-237006ec5058-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7406f81-87cd-42a1-b4bd-237006ec5058-routing-overlay-volume\") pod \"f7406f81-87cd-42a1-b4bd-237006ec5058\" (UID: \"f7406f81-87cd-42a1-b4bd-237006ec5058\") " Oct 30 16:35:54.761354 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:54.761309 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7406f81-87cd-42a1-b4bd-237006ec5058-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7406f81-87cd-42a1-b4bd-237006ec5058-plugins-config-volume\") pod \"f7406f81-87cd-42a1-b4bd-237006ec5058\" (UID: \"f7406f81-87cd-42a1-b4bd-237006ec5058\") " Oct 30 16:35:54.761488 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:54.761387 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7406f81-87cd-42a1-b4bd-237006ec5058-kube-api-access-8mqwg\" (UniqueName: \"kubernetes.io/projected/f7406f81-87cd-42a1-b4bd-237006ec5058-kube-api-access-8mqwg\") pod \"f7406f81-87cd-42a1-b4bd-237006ec5058\" (UID: \"f7406f81-87cd-42a1-b4bd-237006ec5058\") " Oct 30 16:35:54.763003 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:54.762940 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7406f81-87cd-42a1-b4bd-237006ec5058-routing-overlay-volume" pod "f7406f81-87cd-42a1-b4bd-237006ec5058" (UID: "f7406f81-87cd-42a1-b4bd-237006ec5058"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:54.763003 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:54.762952 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7406f81-87cd-42a1-b4bd-237006ec5058-plugins-config-volume" pod "f7406f81-87cd-42a1-b4bd-237006ec5058" (UID: "f7406f81-87cd-42a1-b4bd-237006ec5058"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:35:54.786761 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:54.771481 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7406f81-87cd-42a1-b4bd-237006ec5058-provider-credentials" pod "f7406f81-87cd-42a1-b4bd-237006ec5058" (UID: "f7406f81-87cd-42a1-b4bd-237006ec5058"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:54.786761 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:54.777615 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7406f81-87cd-42a1-b4bd-237006ec5058-kube-api-access-8mqwg" pod "f7406f81-87cd-42a1-b4bd-237006ec5058" (UID: "f7406f81-87cd-42a1-b4bd-237006ec5058"). InnerVolumeSpecName "kube-api-access-8mqwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:35:54.871178 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:54.868573 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7406f81-87cd-42a1-b4bd-237006ec5058-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:35:54.871178 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:54.868612 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7406f81-87cd-42a1-b4bd-237006ec5058-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:35:54.872621 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:54.868628 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7406f81-87cd-42a1-b4bd-237006ec5058-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:35:54.872621 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:54.872064 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mqwg\" (UniqueName: \"kubernetes.io/projected/f7406f81-87cd-42a1-b4bd-237006ec5058-kube-api-access-8mqwg\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:35:55.171759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:55.162820 4861 generic.go:356] "Generic (PLEG): container finished" podID="f7406f81-87cd-42a1-b4bd-237006ec5058" containerID="20bc5d701cdb3f7c70fe78452e5d08a9aac3624ea048227b7db7a407df1007b3" exitCode=0 Oct 30 16:35:55.171759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:55.162895 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" event={"ID":"f7406f81-87cd-42a1-b4bd-237006ec5058","Type":"ContainerDied","Data":"20bc5d701cdb3f7c70fe78452e5d08a9aac3624ea048227b7db7a407df1007b3"} Oct 30 16:35:55.171759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:55.162963 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" event={"ID":"f7406f81-87cd-42a1-b4bd-237006ec5058","Type":"ContainerDied","Data":"8fb8480b201833b85acc8c5a08e853c79b21b264a96170dd689905c66c54c809"} Oct 30 16:35:55.171759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:55.162998 4861 scope.go:122] "RemoveContainer" containerID="20bc5d701cdb3f7c70fe78452e5d08a9aac3624ea048227b7db7a407df1007b3" Oct 30 16:35:55.171759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:55.163160 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w" Oct 30 16:35:55.195252 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:55.195232 4861 scope.go:122] "RemoveContainer" containerID="20bc5d701cdb3f7c70fe78452e5d08a9aac3624ea048227b7db7a407df1007b3" Oct 30 16:35:55.195479 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:35:55.195457 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20bc5d701cdb3f7c70fe78452e5d08a9aac3624ea048227b7db7a407df1007b3\": container with ID starting with 20bc5d701cdb3f7c70fe78452e5d08a9aac3624ea048227b7db7a407df1007b3 not found: ID does not exist" containerID="20bc5d701cdb3f7c70fe78452e5d08a9aac3624ea048227b7db7a407df1007b3" Oct 30 16:35:55.195543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:55.195488 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20bc5d701cdb3f7c70fe78452e5d08a9aac3624ea048227b7db7a407df1007b3"} err="failed to get container status \"20bc5d701cdb3f7c70fe78452e5d08a9aac3624ea048227b7db7a407df1007b3\": rpc error: code = NotFound desc = could not find container \"20bc5d701cdb3f7c70fe78452e5d08a9aac3624ea048227b7db7a407df1007b3\": container with ID starting with 20bc5d701cdb3f7c70fe78452e5d08a9aac3624ea048227b7db7a407df1007b3 not found: ID does not exist" Oct 30 16:35:55.256757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:55.253364 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w"] Oct 30 16:35:55.266188 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:55.266157 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-795bdd7876-qxl8w"] Oct 30 16:35:55.810408 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:55.809988 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7406f81-87cd-42a1-b4bd-237006ec5058" path="/var/lib/kubelet/pods/f7406f81-87cd-42a1-b4bd-237006ec5058/volumes" Oct 30 16:35:56.377891 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.377858 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth"] Oct 30 16:35:56.378963 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.378925 4861 cpu_manager.go:414] "containerMap: removing container" podUID="f7406f81-87cd-42a1-b4bd-237006ec5058" containerName="payload-processing" Oct 30 16:35:56.379133 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.379116 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7406f81-87cd-42a1-b4bd-237006ec5058" containerName="payload-processing" Oct 30 16:35:56.379367 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.379352 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7406f81-87cd-42a1-b4bd-237006ec5058" containerName="payload-processing" Oct 30 16:35:56.391297 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.391276 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.411069 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.411035 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth"] Oct 30 16:35:56.419279 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.419256 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f3503b\"/\"openshift-service-ca.crt\"" Oct 30 16:35:56.419607 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.419588 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3503b\"/\"distinct-w0-f3503b-kserve-self-signed-certs\"" Oct 30 16:35:56.420423 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.420403 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3503b\"/\"default-dockercfg-n7c78\"" Oct 30 16:35:56.420685 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.420667 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f3503b\"/\"kube-root-ca.crt\"" Oct 30 16:35:56.488109 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.487490 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:56.496715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.496683 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-home\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.496905 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.496822 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-dshm\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.497069 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.497048 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-model-cache\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.497137 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.497092 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-tmp-dir\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.497193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.497131 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1837e2a6-812b-4200-bee2-21c23c1a8bb9-tls-certs\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.497248 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.497193 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffhm7\" (UniqueName: \"kubernetes.io/projected/1837e2a6-812b-4200-bee2-21c23c1a8bb9-kube-api-access-ffhm7\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.610743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.606464 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffhm7\" (UniqueName: \"kubernetes.io/projected/1837e2a6-812b-4200-bee2-21c23c1a8bb9-kube-api-access-ffhm7\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.610743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.606528 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-home\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.610743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.606606 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-dshm\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.610743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.606752 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-model-cache\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.610743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.606787 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-tmp-dir\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.610743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.606824 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1837e2a6-812b-4200-bee2-21c23c1a8bb9-tls-certs\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.610743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.608280 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-home\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.610743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.608533 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-model-cache\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.610743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.608831 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-tmp-dir\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.642202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.639404 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1837e2a6-812b-4200-bee2-21c23c1a8bb9-tls-certs\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.642857 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.642829 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:35:56.654758 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.645501 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffhm7\" (UniqueName: \"kubernetes.io/projected/1837e2a6-812b-4200-bee2-21c23c1a8bb9-kube-api-access-ffhm7\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.674850 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.674816 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-dshm\") pod \"distinct-w0-f3503b-kserve-5998b8d978-dwhth\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:56.718752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:56.710940 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:57.113093 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:57.113068 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth"] Oct 30 16:35:57.114482 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:35:57.114449 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1837e2a6_812b_4200_bee2_21c23c1a8bb9.slice/crio-989b5a9c13699f295829abeb4a8383f1db6ad0d4fdb25b2b72ae6436a2b4c0cf WatchSource:0}: Error finding container 989b5a9c13699f295829abeb4a8383f1db6ad0d4fdb25b2b72ae6436a2b4c0cf: Status 404 returned error can't find the container with id 989b5a9c13699f295829abeb4a8383f1db6ad0d4fdb25b2b72ae6436a2b4c0cf Oct 30 16:35:57.190450 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:57.190420 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" event={"ID":"1837e2a6-812b-4200-bee2-21c23c1a8bb9","Type":"ContainerStarted","Data":"2520f862fc2eca9835531159b47679229aaef76f3394e498691b3be6bfe4a74d"} Oct 30 16:35:57.190581 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:57.190466 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" event={"ID":"1837e2a6-812b-4200-bee2-21c23c1a8bb9","Type":"ContainerStarted","Data":"989b5a9c13699f295829abeb4a8383f1db6ad0d4fdb25b2b72ae6436a2b4c0cf"} Oct 30 16:35:57.237504 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:57.237456 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podStartSLOduration=1.237442227 podStartE2EDuration="1.237442227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:57.235941343 +0000 UTC m=+787.578615381" watchObservedRunningTime="2026-10-30 16:35:57.237442227 +0000 UTC m=+787.580116264" Oct 30 16:35:57.712615 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:57.712584 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:35:58.503115 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:35:58.503083 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:00.305016 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:00.304984 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:02.102704 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:02.102674 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:02.819748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:02.819364 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:03.132430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:03.130694 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:36:03.276477 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:03.276446 4861 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-503ee2727cb9: secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:03.277535 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:03.277500 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh: secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:03.277817 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:03.277803 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials podName:225f7152-a7d6-4b53-b705-ad3ea3abf1c9 nodeName:}" failed. No retries permitted until 2026-10-30 16:36:03.777780395 +0000 UTC m=+794.120454426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials") pod "payload-processing-external-model-7cb69f8d98-mb9vh" (UID: "225f7152-a7d6-4b53-b705-ad3ea3abf1c9") : secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:03.873691 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:03.873658 4861 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-503ee2727cb9: secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:03.873922 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:03.873907 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh: secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:03.874110 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:03.874096 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials podName:225f7152-a7d6-4b53-b705-ad3ea3abf1c9 nodeName:}" failed. No retries permitted until 2026-10-30 16:36:04.87407407 +0000 UTC m=+795.216748106 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials") pod "payload-processing-external-model-7cb69f8d98-mb9vh" (UID: "225f7152-a7d6-4b53-b705-ad3ea3abf1c9") : secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:04.310396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.310350 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:04.480073 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.480037 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb"] Oct 30 16:36:04.485017 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.484983 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:04.524014 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.523987 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a169ff0f8c4c\"" Oct 30 16:36:04.537746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.525194 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb"] Oct 30 16:36:04.589970 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.583392 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krhgj\" (UniqueName: \"kubernetes.io/projected/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-kube-api-access-krhgj\") pod \"payload-processing-external-model-789dd89c9f-9vxdb\" (UID: \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\") " pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:04.589970 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.589874 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-routing-overlay-volume\") pod \"payload-processing-external-model-789dd89c9f-9vxdb\" (UID: \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\") " pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:04.590193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.590007 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-plugins-config-volume\") pod \"payload-processing-external-model-789dd89c9f-9vxdb\" (UID: \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\") " pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:04.590193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.590091 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-provider-credentials\") pod \"payload-processing-external-model-789dd89c9f-9vxdb\" (UID: \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\") " pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:04.644015 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.643982 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:04.692082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.692051 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-provider-credentials\") pod \"payload-processing-external-model-789dd89c9f-9vxdb\" (UID: \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\") " pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:04.692577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.692545 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krhgj\" (UniqueName: \"kubernetes.io/projected/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-kube-api-access-krhgj\") pod \"payload-processing-external-model-789dd89c9f-9vxdb\" (UID: \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\") " pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:04.692763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.692747 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-routing-overlay-volume\") pod \"payload-processing-external-model-789dd89c9f-9vxdb\" (UID: \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\") " pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:04.693303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.693281 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-routing-overlay-volume\") pod \"payload-processing-external-model-789dd89c9f-9vxdb\" (UID: \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\") " pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:04.693440 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.693422 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-plugins-config-volume\") pod \"payload-processing-external-model-789dd89c9f-9vxdb\" (UID: \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\") " pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:04.694001 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.693980 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-plugins-config-volume\") pod \"payload-processing-external-model-789dd89c9f-9vxdb\" (UID: \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\") " pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:04.694702 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.694682 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-provider-credentials\") pod \"payload-processing-external-model-789dd89c9f-9vxdb\" (UID: \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\") " pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:04.780142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.780107 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krhgj\" (UniqueName: \"kubernetes.io/projected/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-kube-api-access-krhgj\") pod \"payload-processing-external-model-789dd89c9f-9vxdb\" (UID: \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\") " pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:04.803979 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:04.800953 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:04.921900 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:04.921871 4861 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-503ee2727cb9: secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:04.922087 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:04.922071 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh: secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:04.922292 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:04.922266 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials podName:225f7152-a7d6-4b53-b705-ad3ea3abf1c9 nodeName:}" failed. No retries permitted until 2026-10-30 16:36:06.922241685 +0000 UTC m=+797.264915715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials") pod "payload-processing-external-model-7cb69f8d98-mb9vh" (UID: "225f7152-a7d6-4b53-b705-ad3ea3abf1c9") : secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:05.288319 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:05.288293 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb"] Oct 30 16:36:05.291583 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:36:05.291558 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a265c05_efb3_4ac5_8a5d_1a744eb5c6b9.slice/crio-a124970b2d0595b4a3a2eae290c9cbc8d1f8a036ec2001ad0189c3da649d2d61 WatchSource:0}: Error finding container a124970b2d0595b4a3a2eae290c9cbc8d1f8a036ec2001ad0189c3da649d2d61: Status 404 returned error can't find the container with id a124970b2d0595b4a3a2eae290c9cbc8d1f8a036ec2001ad0189c3da649d2d61 Oct 30 16:36:05.377016 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:05.376939 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" event={"ID":"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9","Type":"ContainerStarted","Data":"a124970b2d0595b4a3a2eae290c9cbc8d1f8a036ec2001ad0189c3da649d2d61"} Oct 30 16:36:06.321870 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:06.312545 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:06.385326 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:06.385281 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" event={"ID":"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9","Type":"ContainerStarted","Data":"e9db4a5187824bde96e10d7b332cfc28c327a36f5a936408b83a2e149b20e61d"} Oct 30 16:36:06.385717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:06.385436 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:06.431064 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:06.431018 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" podStartSLOduration=2.431001723 podStartE2EDuration="2.431001723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:06.428771135 +0000 UTC m=+796.771445174" watchObservedRunningTime="2026-10-30 16:36:06.431001723 +0000 UTC m=+796.773675760" Oct 30 16:36:06.713298 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:06.713215 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:36:06.726235 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:06.726213 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:36:07.006276 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:07.006239 4861 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-503ee2727cb9: secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:07.006276 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:07.006265 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh: secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:07.006529 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:07.006324 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials podName:225f7152-a7d6-4b53-b705-ad3ea3abf1c9 nodeName:}" failed. No retries permitted until 2026-10-30 16:36:11.006306547 +0000 UTC m=+801.348980571 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials") pod "payload-processing-external-model-7cb69f8d98-mb9vh" (UID: "225f7152-a7d6-4b53-b705-ad3ea3abf1c9") : secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:07.401388 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:07.401310 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:36:08.147563 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:08.147528 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:09.997491 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:09.997458 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:11.055278 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:11.055242 4861 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-503ee2727cb9: secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:11.055278 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:11.055269 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh: secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:11.055849 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:11.055328 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials podName:225f7152-a7d6-4b53-b705-ad3ea3abf1c9 nodeName:}" failed. No retries permitted until 2026-10-30 16:36:19.055311147 +0000 UTC m=+809.397985179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials") pod "payload-processing-external-model-7cb69f8d98-mb9vh" (UID: "225f7152-a7d6-4b53-b705-ad3ea3abf1c9") : secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:11.801547 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:11.801514 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:13.542062 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:13.542020 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:15.312074 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:15.312034 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:16.005352 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:16.005307 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:17.393359 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:17.393333 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:17.439338 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:17.439310 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:17.512889 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:17.512846 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh"] Oct 30 16:36:19.138329 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:19.138298 4861 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-503ee2727cb9: secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:19.138329 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:19.138324 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh: secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:19.138856 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:19.138399 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials podName:225f7152-a7d6-4b53-b705-ad3ea3abf1c9 nodeName:}" failed. No retries permitted until 2026-10-30 16:36:35.138381116 +0000 UTC m=+825.481055132 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials") pod "payload-processing-external-model-7cb69f8d98-mb9vh" (UID: "225f7152-a7d6-4b53-b705-ad3ea3abf1c9") : secret "agc-cred-503ee2727cb9" not found Oct 30 16:36:19.399855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:19.399781 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:20.289125 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:20.289092 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:36:21.317525 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:21.317495 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:22.515682 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:22.515640 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" podUID="225f7152-a7d6-4b53-b705-ad3ea3abf1c9" containerName="payload-processing" containerID="cri-o://aa756f127147fef698470df35eb6ead5713bea2840ef8374c77a2badf4c885e5" gracePeriod=25 Oct 30 16:36:22.767135 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:22.767077 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:36:22.790130 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:22.790109 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-routing-overlay-volume\") pod \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\" (UID: \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\") " Oct 30 16:36:22.790441 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:22.790418 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-routing-overlay-volume" pod "225f7152-a7d6-4b53-b705-ad3ea3abf1c9" (UID: "225f7152-a7d6-4b53-b705-ad3ea3abf1c9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:22.891469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:22.891444 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials\") pod \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\" (UID: \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\") " Oct 30 16:36:22.891625 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:22.891484 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-kube-api-access-8jpd6\" (UniqueName: \"kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-kube-api-access-8jpd6\") pod \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\" (UID: \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\") " Oct 30 16:36:22.891625 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:22.891527 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-plugins-config-volume\") pod \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\" (UID: \"225f7152-a7d6-4b53-b705-ad3ea3abf1c9\") " Oct 30 16:36:22.892003 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:22.891970 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:36:22.892129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:22.892009 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-plugins-config-volume" pod "225f7152-a7d6-4b53-b705-ad3ea3abf1c9" (UID: "225f7152-a7d6-4b53-b705-ad3ea3abf1c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:22.893771 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:22.893748 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-kube-api-access-8jpd6" pod "225f7152-a7d6-4b53-b705-ad3ea3abf1c9" (UID: "225f7152-a7d6-4b53-b705-ad3ea3abf1c9"). InnerVolumeSpecName "kube-api-access-8jpd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:22.893864 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:22.893765 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials" pod "225f7152-a7d6-4b53-b705-ad3ea3abf1c9" (UID: "225f7152-a7d6-4b53-b705-ad3ea3abf1c9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:22.993306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:22.993276 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:36:22.993306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:22.993301 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jpd6\" (UniqueName: \"kubernetes.io/projected/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-kube-api-access-8jpd6\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:36:22.993306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:22.993311 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225f7152-a7d6-4b53-b705-ad3ea3abf1c9-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:36:23.196809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:23.196732 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:23.549060 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:23.549027 4861 generic.go:356] "Generic (PLEG): container finished" podID="225f7152-a7d6-4b53-b705-ad3ea3abf1c9" containerID="aa756f127147fef698470df35eb6ead5713bea2840ef8374c77a2badf4c885e5" exitCode=0 Oct 30 16:36:23.549543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:23.549101 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" Oct 30 16:36:23.549543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:23.549106 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" event={"ID":"225f7152-a7d6-4b53-b705-ad3ea3abf1c9","Type":"ContainerDied","Data":"aa756f127147fef698470df35eb6ead5713bea2840ef8374c77a2badf4c885e5"} Oct 30 16:36:23.549543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:23.549210 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh" event={"ID":"225f7152-a7d6-4b53-b705-ad3ea3abf1c9","Type":"ContainerDied","Data":"1fd45b779356d54d132bed9b9f2065247010f7b54f6ef0e4b5886437c0425e79"} Oct 30 16:36:23.549543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:23.549238 4861 scope.go:122] "RemoveContainer" containerID="aa756f127147fef698470df35eb6ead5713bea2840ef8374c77a2badf4c885e5" Oct 30 16:36:23.559476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:23.559459 4861 scope.go:122] "RemoveContainer" containerID="aa756f127147fef698470df35eb6ead5713bea2840ef8374c77a2badf4c885e5" Oct 30 16:36:23.559745 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:23.559708 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa756f127147fef698470df35eb6ead5713bea2840ef8374c77a2badf4c885e5\": container with ID starting with aa756f127147fef698470df35eb6ead5713bea2840ef8374c77a2badf4c885e5 not found: ID does not exist" containerID="aa756f127147fef698470df35eb6ead5713bea2840ef8374c77a2badf4c885e5" Oct 30 16:36:23.559814 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:23.559759 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa756f127147fef698470df35eb6ead5713bea2840ef8374c77a2badf4c885e5"} err="failed to get container status \"aa756f127147fef698470df35eb6ead5713bea2840ef8374c77a2badf4c885e5\": rpc error: code = NotFound desc = could not find container \"aa756f127147fef698470df35eb6ead5713bea2840ef8374c77a2badf4c885e5\": container with ID starting with aa756f127147fef698470df35eb6ead5713bea2840ef8374c77a2badf4c885e5 not found: ID does not exist" Oct 30 16:36:23.603300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:23.603272 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh"] Oct 30 16:36:23.616224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:23.616197 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cb69f8d98-mb9vh"] Oct 30 16:36:23.811810 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:23.811155 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="225f7152-a7d6-4b53-b705-ad3ea3abf1c9" path="/var/lib/kubelet/pods/225f7152-a7d6-4b53-b705-ad3ea3abf1c9/volumes" Oct 30 16:36:25.176393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:25.176364 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:26.953588 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:26.953557 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq"] Oct 30 16:36:26.954378 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:26.954357 4861 cpu_manager.go:414] "containerMap: removing container" podUID="225f7152-a7d6-4b53-b705-ad3ea3abf1c9" containerName="payload-processing" Oct 30 16:36:26.954469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:26.954380 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="225f7152-a7d6-4b53-b705-ad3ea3abf1c9" containerName="payload-processing" Oct 30 16:36:26.954556 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:26.954542 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="225f7152-a7d6-4b53-b705-ad3ea3abf1c9" containerName="payload-processing" Oct 30 16:36:26.959344 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:26.959321 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:26.990458 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:26.990432 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f3503b\"/\"distinct-2-w0-f3503b-kserve-self-signed-certs\"" Oct 30 16:36:26.993112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:26.993087 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq"] Oct 30 16:36:27.012053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.012024 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:27.043063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.043024 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-model-cache\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.043239 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.043219 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-home\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.043286 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.043252 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnv6c\" (UniqueName: \"kubernetes.io/projected/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-kube-api-access-bnv6c\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.043357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.043344 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-dshm\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.043409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.043373 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-tmp-dir\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.043409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.043393 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-tls-certs\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.162523 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.149511 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-model-cache\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.176565 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.163302 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:27.176845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.164212 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-model-cache\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.177337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.177088 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-home\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.177337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.177147 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnv6c\" (UniqueName: \"kubernetes.io/projected/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-kube-api-access-bnv6c\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.177520 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.177446 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-dshm\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.177520 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.177493 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-tmp-dir\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.177635 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.177541 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-tls-certs\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.178667 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.178642 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-home\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.179306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.179285 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-tmp-dir\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.222389 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.222354 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-dshm\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.226854 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.223345 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-tls-certs\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.227521 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.227496 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnv6c\" (UniqueName: \"kubernetes.io/projected/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-kube-api-access-bnv6c\") pod \"distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.273115 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.273077 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:27.580026 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.580000 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq"] Oct 30 16:36:27.581976 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:36:27.581952 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b2d86a2_ddd1_4070_9b52_f6b05d9e38bc.slice/crio-46beef71e434a95c2f9426de50dcaa28585aeeb88d3e77cf252207160b90bed1 WatchSource:0}: Error finding container 46beef71e434a95c2f9426de50dcaa28585aeeb88d3e77cf252207160b90bed1: Status 404 returned error can't find the container with id 46beef71e434a95c2f9426de50dcaa28585aeeb88d3e77cf252207160b90bed1 Oct 30 16:36:27.822319 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.822292 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j"] Oct 30 16:36:27.826297 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.826266 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.853365 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.853334 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j"] Oct 30 16:36:27.884615 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.884575 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.884615 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.884617 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.884855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.884710 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.884855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.884837 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrqxj\" (UniqueName: \"kubernetes.io/projected/fa8794c5-5a39-47d8-b870-c7cf40526fb6-kube-api-access-hrqxj\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.884975 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.884859 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.884975 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.884909 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.884975 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.884929 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.885175 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.885157 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.885215 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.885192 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.986889 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.986857 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.987347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.986898 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.987347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.987065 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.987347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.987125 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrqxj\" (UniqueName: \"kubernetes.io/projected/fa8794c5-5a39-47d8-b870-c7cf40526fb6-kube-api-access-hrqxj\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.987347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.987162 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.987347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.987182 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.987347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.987202 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.987347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.987298 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.987347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.987341 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.987812 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.987380 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.987812 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.987772 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.988098 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.987881 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.988098 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.987925 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.988098 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.988021 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.989706 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.989683 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:27.989892 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:27.989873 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:28.013070 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:28.013041 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrqxj\" (UniqueName: \"kubernetes.io/projected/fa8794c5-5a39-47d8-b870-c7cf40526fb6-kube-api-access-hrqxj\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:28.013598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:28.013575 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:28.140311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:28.140225 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:28.467536 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:28.467508 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j"] Oct 30 16:36:28.468147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:36:28.468120 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8794c5_5a39_47d8_b870_c7cf40526fb6.slice/crio-17d101e0d071ae9397d2cf569c754548854e1cd6053e34f52782e259b2909050 WatchSource:0}: Error finding container 17d101e0d071ae9397d2cf569c754548854e1cd6053e34f52782e259b2909050: Status 404 returned error can't find the container with id 17d101e0d071ae9397d2cf569c754548854e1cd6053e34f52782e259b2909050 Oct 30 16:36:28.470592 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:28.470565 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:36:28.470663 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:28.470629 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:36:28.470698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:28.470661 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:36:28.585455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:28.585416 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" event={"ID":"fa8794c5-5a39-47d8-b870-c7cf40526fb6","Type":"ContainerStarted","Data":"7deddd04b344f12ed4242a8df624f5df1ac72ccfac4db9629b2b06cfb556c656"} Oct 30 16:36:28.585624 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:28.585482 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" event={"ID":"fa8794c5-5a39-47d8-b870-c7cf40526fb6","Type":"ContainerStarted","Data":"17d101e0d071ae9397d2cf569c754548854e1cd6053e34f52782e259b2909050"} Oct 30 16:36:28.587375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:28.587346 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" event={"ID":"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc","Type":"ContainerStarted","Data":"1f5cb58ea1fbe3687ea2af5157c1a3596dbac64700ef488a5fc312eef4539552"} Oct 30 16:36:28.587506 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:28.587385 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" event={"ID":"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc","Type":"ContainerStarted","Data":"46beef71e434a95c2f9426de50dcaa28585aeeb88d3e77cf252207160b90bed1"} Oct 30 16:36:28.634310 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:28.634259 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" podStartSLOduration=1.634244537 podStartE2EDuration="1.634244537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:28.631889269 +0000 UTC m=+818.974563307" watchObservedRunningTime="2026-10-30 16:36:28.634244537 +0000 UTC m=+818.976918575" Oct 30 16:36:28.675620 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:28.675575 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podStartSLOduration=2.6755605 podStartE2EDuration="2.6755605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:28.674318762 +0000 UTC m=+819.016992804" watchObservedRunningTime="2026-10-30 16:36:28.6755605 +0000 UTC m=+819.018234538" Oct 30 16:36:28.907577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:28.907533 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:29.141396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.141360 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:29.143334 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.143307 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 16:36:29.143496 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.143364 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" podUID="fa8794c5-5a39-47d8-b870-c7cf40526fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 16:36:29.182042 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.181973 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb"] Oct 30 16:36:29.185959 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.185941 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.241903 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.241870 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb"] Oct 30 16:36:29.273973 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.273945 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:29.341678 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.341643 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dwzg\" (UniqueName: \"kubernetes.io/projected/6abf00fa-1784-428a-9ddc-afa6963b2985-kube-api-access-8dwzg\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.341895 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.341713 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.341962 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.341889 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.341962 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.341927 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6abf00fa-1784-428a-9ddc-afa6963b2985-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.342082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.342016 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.342082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.342043 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.342177 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.342104 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.342177 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.342162 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.342284 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.342182 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.443394 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.443304 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dwzg\" (UniqueName: \"kubernetes.io/projected/6abf00fa-1784-428a-9ddc-afa6963b2985-kube-api-access-8dwzg\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.443394 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.443382 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.443641 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.443440 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.443641 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.443475 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6abf00fa-1784-428a-9ddc-afa6963b2985-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.443641 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.443507 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.443821 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.443658 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.443821 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.443710 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.443821 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.443769 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.443821 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.443802 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.444026 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.443880 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.444078 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.444028 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.444241 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.444204 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.444331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.444242 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.444853 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.444828 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6abf00fa-1784-428a-9ddc-afa6963b2985-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.445935 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.445914 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.446200 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.446183 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.467627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.467592 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.467777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.467760 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dwzg\" (UniqueName: \"kubernetes.io/projected/6abf00fa-1784-428a-9ddc-afa6963b2985-kube-api-access-8dwzg\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.518748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.514967 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:29.855443 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.855397 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb"] Oct 30 16:36:29.857895 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:36:29.857869 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6abf00fa_1784_428a_9ddc_afa6963b2985.slice/crio-55b398b7fe7a82f22f25b7b82058b985d77272cb5f3eb7887e8448e9ba09c662 WatchSource:0}: Error finding container 55b398b7fe7a82f22f25b7b82058b985d77272cb5f3eb7887e8448e9ba09c662: Status 404 returned error can't find the container with id 55b398b7fe7a82f22f25b7b82058b985d77272cb5f3eb7887e8448e9ba09c662 Oct 30 16:36:29.860290 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.860262 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:36:29.860373 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.860318 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:36:29.860373 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:29.860344 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:36:30.140801 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:30.140558 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 16:36:30.140801 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:30.140616 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" podUID="fa8794c5-5a39-47d8-b870-c7cf40526fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 16:36:30.619950 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:30.619904 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" event={"ID":"6abf00fa-1784-428a-9ddc-afa6963b2985","Type":"ContainerStarted","Data":"46784ebd12399423ca46aa4f2bfbdc0ab34564f66c3e12ae92026511ac88196f"} Oct 30 16:36:30.620453 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:30.620029 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" event={"ID":"6abf00fa-1784-428a-9ddc-afa6963b2985","Type":"ContainerStarted","Data":"55b398b7fe7a82f22f25b7b82058b985d77272cb5f3eb7887e8448e9ba09c662"} Oct 30 16:36:30.679092 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:30.679041 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" podStartSLOduration=1.67902695 podStartE2EDuration="1.67902695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:30.677898354 +0000 UTC m=+821.020572397" watchObservedRunningTime="2026-10-30 16:36:30.67902695 +0000 UTC m=+821.021701053" Oct 30 16:36:30.822403 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:30.822366 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:31.197754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:31.194118 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 16:36:31.197754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:31.194185 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" podUID="fa8794c5-5a39-47d8-b870-c7cf40526fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 16:36:31.223745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:31.198573 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb"] Oct 30 16:36:31.279958 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:31.279921 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:31.454206 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:31.454079 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:31.529509 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:31.529478 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:31.529913 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:31.529891 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:36:31.530222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:31.530171 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" podUID="6abf00fa-1784-428a-9ddc-afa6963b2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 16:36:32.146750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:32.145830 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 16:36:32.146750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:32.145892 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" podUID="fa8794c5-5a39-47d8-b870-c7cf40526fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 16:36:32.525755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:32.524151 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:36:32.525755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:32.524212 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" podUID="6abf00fa-1784-428a-9ddc-afa6963b2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 16:36:32.938800 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:32.938532 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:33.146405 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:33.144927 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 16:36:33.146405 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:33.145027 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" podUID="fa8794c5-5a39-47d8-b870-c7cf40526fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 16:36:33.524484 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:33.524072 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:36:33.524484 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:33.524131 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" podUID="6abf00fa-1784-428a-9ddc-afa6963b2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 16:36:34.147749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:34.147384 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 16:36:34.147749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:34.147440 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" podUID="fa8794c5-5a39-47d8-b870-c7cf40526fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 16:36:34.516747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:34.516165 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:36:34.516747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:34.516230 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" podUID="6abf00fa-1784-428a-9ddc-afa6963b2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 16:36:35.010046 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:35.009994 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:35.148782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:35.145823 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 16:36:35.148782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:35.145879 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" podUID="fa8794c5-5a39-47d8-b870-c7cf40526fb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 16:36:35.518288 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:35.518250 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:36:35.518633 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:35.518600 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" podUID="6abf00fa-1784-428a-9ddc-afa6963b2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 16:36:36.144864 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.144833 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:36.145348 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.145326 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:36.146059 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.146039 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 16:36:36.199811 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.199783 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" podUID="0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9" containerName="payload-processing" containerID="cri-o://e9db4a5187824bde96e10d7b332cfc28c327a36f5a936408b83a2e149b20e61d" gracePeriod=25 Oct 30 16:36:36.517529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.517194 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:36:36.517529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.517288 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" podUID="6abf00fa-1784-428a-9ddc-afa6963b2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 16:36:36.549412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.549384 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:36.698524 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.698428 4861 generic.go:356] "Generic (PLEG): container finished" podID="0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9" containerID="e9db4a5187824bde96e10d7b332cfc28c327a36f5a936408b83a2e149b20e61d" exitCode=0 Oct 30 16:36:36.698524 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.698477 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" Oct 30 16:36:36.698524 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.698511 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" event={"ID":"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9","Type":"ContainerDied","Data":"e9db4a5187824bde96e10d7b332cfc28c327a36f5a936408b83a2e149b20e61d"} Oct 30 16:36:36.698817 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.698560 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb" event={"ID":"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9","Type":"ContainerDied","Data":"a124970b2d0595b4a3a2eae290c9cbc8d1f8a036ec2001ad0189c3da649d2d61"} Oct 30 16:36:36.698817 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.698581 4861 scope.go:122] "RemoveContainer" containerID="e9db4a5187824bde96e10d7b332cfc28c327a36f5a936408b83a2e149b20e61d" Oct 30 16:36:36.702153 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.702133 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-plugins-config-volume\") pod \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\" (UID: \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\") " Oct 30 16:36:36.702246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.702183 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-kube-api-access-krhgj\" (UniqueName: \"kubernetes.io/projected/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-kube-api-access-krhgj\") pod \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\" (UID: \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\") " Oct 30 16:36:36.702246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.702221 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-provider-credentials\") pod \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\" (UID: \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\") " Oct 30 16:36:36.702246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.702245 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-routing-overlay-volume\") pod \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\" (UID: \"0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9\") " Oct 30 16:36:36.702670 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.702635 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-plugins-config-volume" pod "0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9" (UID: "0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:36.702829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.702709 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-routing-overlay-volume" pod "0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9" (UID: "0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:36.704698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.704671 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-kube-api-access-krhgj" pod "0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9" (UID: "0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9"). InnerVolumeSpecName "kube-api-access-krhgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:36.704698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.704681 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-provider-credentials" pod "0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9" (UID: "0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:36.708546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.708524 4861 scope.go:122] "RemoveContainer" containerID="e9db4a5187824bde96e10d7b332cfc28c327a36f5a936408b83a2e149b20e61d" Oct 30 16:36:36.708882 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:36.708860 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9db4a5187824bde96e10d7b332cfc28c327a36f5a936408b83a2e149b20e61d\": container with ID starting with e9db4a5187824bde96e10d7b332cfc28c327a36f5a936408b83a2e149b20e61d not found: ID does not exist" containerID="e9db4a5187824bde96e10d7b332cfc28c327a36f5a936408b83a2e149b20e61d" Oct 30 16:36:36.708982 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.708888 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9db4a5187824bde96e10d7b332cfc28c327a36f5a936408b83a2e149b20e61d"} err="failed to get container status \"e9db4a5187824bde96e10d7b332cfc28c327a36f5a936408b83a2e149b20e61d\": rpc error: code = NotFound desc = could not find container \"e9db4a5187824bde96e10d7b332cfc28c327a36f5a936408b83a2e149b20e61d\": container with ID starting with e9db4a5187824bde96e10d7b332cfc28c327a36f5a936408b83a2e149b20e61d not found: ID does not exist" Oct 30 16:36:36.804689 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.804529 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:36:36.804689 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.804565 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krhgj\" (UniqueName: \"kubernetes.io/projected/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-kube-api-access-krhgj\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:36:36.804689 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.804582 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:36:36.804689 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.804599 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:36:36.917678 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:36.917597 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:37.051217 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:37.051186 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb"] Oct 30 16:36:37.054965 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:37.054942 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-789dd89c9f-9vxdb"] Oct 30 16:36:37.273921 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:37.273824 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:37.293756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:37.293709 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:37.717226 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:37.717205 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:36:37.808651 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:37.808619 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9" path="/var/lib/kubelet/pods/0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9/volumes" Oct 30 16:36:38.556911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:38.556872 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:36:38.557387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:38.556938 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" podUID="6abf00fa-1784-428a-9ddc-afa6963b2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:38.743381 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:38.743349 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:38.743980 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:38.743951 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:38.744982 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:38.744960 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:36:38.789570 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:38.789541 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:40.538475 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:40.537812 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:41.465277 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:41.465242 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:42.368250 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:42.368187 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:43.256432 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.256402 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:43.432277 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.432233 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65"] Oct 30 16:36:43.471089 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.471058 4861 cpu_manager.go:414] "containerMap: removing container" podUID="0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9" containerName="payload-processing" Oct 30 16:36:43.471302 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.471288 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9" containerName="payload-processing" Oct 30 16:36:43.471608 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.471570 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a265c05-efb3-4ac5-8a5d-1a744eb5c6b9" containerName="payload-processing" Oct 30 16:36:43.478236 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.477316 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65"] Oct 30 16:36:43.478236 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.477428 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.510658 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.510603 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.510775 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.510657 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c3c5184-e45a-4301-af8f-85e8003a171a-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.510841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.510821 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.510886 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.510873 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.510962 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.510946 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.511029 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.511014 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxts\" (UniqueName: \"kubernetes.io/projected/4c3c5184-e45a-4301-af8f-85e8003a171a-kube-api-access-gbxts\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.511067 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.511051 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.511104 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.511073 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.511104 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.511093 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.611834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.611795 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c3c5184-e45a-4301-af8f-85e8003a171a-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.611980 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.611870 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.611980 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.611906 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.611980 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.611948 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.612134 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.611998 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxts\" (UniqueName: \"kubernetes.io/projected/4c3c5184-e45a-4301-af8f-85e8003a171a-kube-api-access-gbxts\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.612134 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.612027 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.612258 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.612225 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.612322 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.612295 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.612376 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.612338 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.612376 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.612356 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.612376 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.612362 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.612544 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.612518 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c3c5184-e45a-4301-af8f-85e8003a171a-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.612617 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.612596 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.612617 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.612610 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.614492 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.614468 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.614783 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.614765 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.635999 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.635978 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.636489 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.636473 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxts\" (UniqueName: \"kubernetes.io/projected/4c3c5184-e45a-4301-af8f-85e8003a171a-kube-api-access-gbxts\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:43.799274 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:43.799178 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:44.128877 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:36:44.128850 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c3c5184_e45a_4301_af8f_85e8003a171a.slice/crio-098683ba16268aabbbb3ad0fdfb76a2846a8b0277ec0a5512cf1951c585ab74b WatchSource:0}: Error finding container 098683ba16268aabbbb3ad0fdfb76a2846a8b0277ec0a5512cf1951c585ab74b: Status 404 returned error can't find the container with id 098683ba16268aabbbb3ad0fdfb76a2846a8b0277ec0a5512cf1951c585ab74b Oct 30 16:36:44.130546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:44.130513 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:36:44.130633 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:44.130591 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:36:44.130677 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:44.130636 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:36:44.134470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:44.134446 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65"] Oct 30 16:36:44.240750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:44.237248 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:44.763304 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:44.763266 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" event={"ID":"4c3c5184-e45a-4301-af8f-85e8003a171a","Type":"ContainerStarted","Data":"e4c9c18baf95be70caec7c463170d00c6c2d54c159afb85ef004b831556b765e"} Oct 30 16:36:44.763691 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:44.763333 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" event={"ID":"4c3c5184-e45a-4301-af8f-85e8003a171a","Type":"ContainerStarted","Data":"098683ba16268aabbbb3ad0fdfb76a2846a8b0277ec0a5512cf1951c585ab74b"} Oct 30 16:36:44.799319 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:44.799288 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:44.836015 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:44.835971 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" podStartSLOduration=1.835957401 podStartE2EDuration="1.835957401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:44.822535593 +0000 UTC m=+835.165209625" watchObservedRunningTime="2026-10-30 16:36:44.835957401 +0000 UTC m=+835.178631439" Oct 30 16:36:45.820644 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:45.820608 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:36:45.821270 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:45.821239 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" podUID="4c3c5184-e45a-4301-af8f-85e8003a171a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 16:36:46.126419 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:46.126331 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:46.800197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:46.800166 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:36:46.800380 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:46.800218 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" podUID="4c3c5184-e45a-4301-af8f-85e8003a171a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 16:36:47.799596 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:47.799564 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:36:47.800030 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:47.799614 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" podUID="4c3c5184-e45a-4301-af8f-85e8003a171a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 16:36:48.126747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:48.126625 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:48.800064 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:48.800034 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:36:48.800456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:48.800084 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" podUID="4c3c5184-e45a-4301-af8f-85e8003a171a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 16:36:49.075168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:49.075074 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:49.810878 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:49.810715 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:36:49.811447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:49.811412 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" podUID="4c3c5184-e45a-4301-af8f-85e8003a171a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 16:36:50.295618 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:36:50.295587 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:36:50.384804 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:50.382521 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:50.800269 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:50.800231 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:36:50.800443 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:50.800299 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" podUID="4c3c5184-e45a-4301-af8f-85e8003a171a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 16:36:51.408951 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.408908 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27"] Oct 30 16:36:51.414644 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.414620 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.446957 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.446845 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27"] Oct 30 16:36:51.451805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.451764 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a7f28a-b91a-4b4b-984d-a390a3d49776-tls-certs\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.451984 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.451951 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-home\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.452137 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.452000 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-model-cache\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.452137 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.452072 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-tmp-dir\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.452283 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.452154 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-dshm\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.452283 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.452198 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5twj\" (UniqueName: \"kubernetes.io/projected/e9a7f28a-b91a-4b4b-984d-a390a3d49776-kube-api-access-l5twj\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.455868 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.455843 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-420ff3\"/\"default-dockercfg-2wjrq\"" Oct 30 16:36:51.455979 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.455918 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-420ff3\"/\"distinct-w4-420ff3-kserve-self-signed-certs\"" Oct 30 16:36:51.456050 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.456021 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-420ff3\"/\"openshift-service-ca.crt\"" Oct 30 16:36:51.456175 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.456154 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-420ff3\"/\"kube-root-ca.crt\"" Oct 30 16:36:51.507610 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.507581 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:51.553661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.553635 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-home\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.553805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.553662 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-model-cache\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.553805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.553699 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-tmp-dir\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.553805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.553760 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-dshm\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.553805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.553786 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5twj\" (UniqueName: \"kubernetes.io/projected/e9a7f28a-b91a-4b4b-984d-a390a3d49776-kube-api-access-l5twj\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.554019 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.553833 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a7f28a-b91a-4b4b-984d-a390a3d49776-tls-certs\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.554136 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.554113 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-home\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.554225 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.554199 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-model-cache\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.554416 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.554373 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-tmp-dir\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.556932 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.556902 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-dshm\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.557489 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.557467 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a7f28a-b91a-4b4b-984d-a390a3d49776-tls-certs\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.577955 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.577930 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5twj\" (UniqueName: \"kubernetes.io/projected/e9a7f28a-b91a-4b4b-984d-a390a3d49776-kube-api-access-l5twj\") pod \"distinct-w4-420ff3-kserve-6497d7466-nhk27\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.731607 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.731556 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:51.929702 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.929335 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:51.930350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.930325 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:51.931539 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:51.931520 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:36:52.056151 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:36:52.056116 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9a7f28a_b91a_4b4b_984d_a390a3d49776.slice/crio-8159f114e2ea9ba18a35081a3d73a5b25b2f74607e9ab4129dc4bf016702539b WatchSource:0}: Error finding container 8159f114e2ea9ba18a35081a3d73a5b25b2f74607e9ab4129dc4bf016702539b: Status 404 returned error can't find the container with id 8159f114e2ea9ba18a35081a3d73a5b25b2f74607e9ab4129dc4bf016702539b Oct 30 16:36:52.063455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:52.063422 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27"] Oct 30 16:36:52.482876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:52.482843 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:52.848968 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:52.848866 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" event={"ID":"e9a7f28a-b91a-4b4b-984d-a390a3d49776","Type":"ContainerStarted","Data":"2ff854f7db9f5933e2e3434b646799b88fe24bcc38ed523357c5a76c95b6e95b"} Oct 30 16:36:52.848968 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:52.848914 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" event={"ID":"e9a7f28a-b91a-4b4b-984d-a390a3d49776","Type":"ContainerStarted","Data":"8159f114e2ea9ba18a35081a3d73a5b25b2f74607e9ab4129dc4bf016702539b"} Oct 30 16:36:52.894641 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:52.894592 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podStartSLOduration=1.894578789 podStartE2EDuration="1.894578789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:52.89151814 +0000 UTC m=+843.234192186" watchObservedRunningTime="2026-10-30 16:36:52.894578789 +0000 UTC m=+843.237252827" Oct 30 16:36:53.732039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:53.732009 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:36:54.290867 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:54.290833 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:56.098240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:56.098196 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:56.381853 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:56.381754 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:57.902519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:57.902488 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:36:59.996744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:36:59.996529 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:01.732606 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:01.732565 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:37:01.745062 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:01.745036 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:37:01.969365 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:01.969335 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:37:02.126870 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:02.126793 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:04.005106 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:04.005078 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:04.173325 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:04.173293 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:05.895586 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:05.895555 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:07.702327 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:07.702273 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:09.494051 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:09.494022 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:11.302679 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:11.302650 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:12.985925 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:12.985878 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8"] Oct 30 16:37:12.990278 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:12.990255 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.018560 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.018528 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8"] Oct 30 16:37:13.056633 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.056591 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9b9641-1b2c-49b7-9169-59cf182f4900-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.056836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.056642 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.056927 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.056873 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w4tj\" (UniqueName: \"kubernetes.io/projected/db9b9641-1b2c-49b7-9169-59cf182f4900-kube-api-access-7w4tj\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.057013 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.056994 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.057073 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.057021 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.057129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.057108 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.057197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.057180 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.057247 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.057211 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.057307 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.057288 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.115095 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.115053 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:13.166629 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.166559 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.166629 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.166605 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.166938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.166639 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.166938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.166679 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.166938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.166702 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.166938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.166756 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.166938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.166863 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9b9641-1b2c-49b7-9169-59cf182f4900-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.166938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.166899 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.167222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.167024 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w4tj\" (UniqueName: \"kubernetes.io/projected/db9b9641-1b2c-49b7-9169-59cf182f4900-kube-api-access-7w4tj\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.167222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.167109 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.167658 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.167114 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.167658 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.167374 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.167916 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.167824 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9b9641-1b2c-49b7-9169-59cf182f4900-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.168395 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.168346 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.170881 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.170859 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.172514 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.172464 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.195015 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.194985 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w4tj\" (UniqueName: \"kubernetes.io/projected/db9b9641-1b2c-49b7-9169-59cf182f4900-kube-api-access-7w4tj\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.195639 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.195611 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.305095 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.305016 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:13.635091 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.635055 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8"] Oct 30 16:37:13.635754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.635677 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:37:13.635896 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.635832 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:37:13.635954 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:13.635923 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:37:14.035885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.035849 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" event={"ID":"db9b9641-1b2c-49b7-9169-59cf182f4900","Type":"ContainerStarted","Data":"0b4459e251555fed4b5e9f90878719cec951a5b8b5b641a7d5c8087eb027da97"} Oct 30 16:37:14.036303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.035924 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" event={"ID":"db9b9641-1b2c-49b7-9169-59cf182f4900","Type":"ContainerStarted","Data":"d8c8140130350aebb7735722d6a2d5c5550533ac387cc08cb6650feec5ad299c"} Oct 30 16:37:14.084148 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.084101 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" podStartSLOduration=2.084087044 podStartE2EDuration="2.084087044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:14.082638132 +0000 UTC m=+864.425312167" watchObservedRunningTime="2026-10-30 16:37:14.084087044 +0000 UTC m=+864.426761082" Oct 30 16:37:14.308565 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.307593 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw"] Oct 30 16:37:14.312747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.311012 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:37:14.312747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.311105 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 16:37:14.321746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.321081 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:14.321746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.321214 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.346739 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.346694 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw"] Oct 30 16:37:14.381787 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.381752 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.381946 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.381799 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbsvj\" (UniqueName: \"kubernetes.io/projected/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-kube-api-access-cbsvj\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.381946 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.381898 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.382029 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.381995 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.382086 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.382060 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.382171 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.382144 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.382209 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.382190 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.382283 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.382267 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.382336 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.382303 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.483850 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.483815 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.484053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.483861 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.484053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.484008 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.484185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.484061 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbsvj\" (UniqueName: \"kubernetes.io/projected/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-kube-api-access-cbsvj\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.484185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.484095 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.484185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.484178 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.484340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.484233 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.484340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.484263 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.484340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.484290 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.484487 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.484371 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.484666 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.484629 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.484666 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.484655 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.484817 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.484680 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.485337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.485312 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.486686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.486654 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.487093 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.487072 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.523394 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.523328 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.532735 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.532682 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbsvj\" (UniqueName: \"kubernetes.io/projected/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-kube-api-access-cbsvj\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.642403 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.642306 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:14.966847 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.966756 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw"] Oct 30 16:37:14.979318 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:37:14.979289 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f90cfb5_ffa8_40ca_af8c_64c24bc3e3ec.slice/crio-78fb9b6e919848e227dafc4911ede46b1b5c4c91944b433268e31204e7ce5034 WatchSource:0}: Error finding container 78fb9b6e919848e227dafc4911ede46b1b5c4c91944b433268e31204e7ce5034: Status 404 returned error can't find the container with id 78fb9b6e919848e227dafc4911ede46b1b5c4c91944b433268e31204e7ce5034 Oct 30 16:37:14.981967 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.981926 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:37:14.982078 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.982014 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:37:14.982078 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:14.982059 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:37:15.041136 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:15.041102 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:15.044925 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:15.044903 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" event={"ID":"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec","Type":"ContainerStarted","Data":"78fb9b6e919848e227dafc4911ede46b1b5c4c91944b433268e31204e7ce5034"} Oct 30 16:37:15.331893 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:15.331855 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:37:15.332069 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:15.331916 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 16:37:16.064374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:16.064333 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" event={"ID":"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec","Type":"ContainerStarted","Data":"71edbdf0be695dfe5025ca9e06b252e945507058e11cb8a0298952e860fef70d"} Oct 30 16:37:16.163785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:16.163005 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" podStartSLOduration=2.16298469 podStartE2EDuration="2.16298469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:16.14316313 +0000 UTC m=+866.485837170" watchObservedRunningTime="2026-10-30 16:37:16.16298469 +0000 UTC m=+866.505658730" Oct 30 16:37:16.318806 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:16.311146 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:37:16.318806 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:16.311221 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 16:37:16.655830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:16.644964 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:16.655830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:16.645083 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:37:16.655830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:16.645126 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" podUID="3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 16:37:17.147827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:17.134745 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:17.308199 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:17.308164 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:37:17.308474 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:17.308445 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 16:37:17.643749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:17.643164 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:37:17.643749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:17.643228 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" podUID="3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 16:37:18.330782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:18.304135 4861 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:18.330782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:18.305849 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:18.358764 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:18.332426 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:37:18.358764 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:18.332481 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 16:37:18.358764 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:18.343792 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:18.394753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:18.394451 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 16:37:18.654804 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:18.644141 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:37:18.654804 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:18.644201 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" podUID="3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 16:37:18.736124 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:18.736063 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:19.030749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:19.029815 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:19.050937 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:19.050887 4861 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:19.308925 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:19.308809 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:19.311116 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:19.309846 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:37:19.311116 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:19.309452 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:37:19.311116 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:19.310626 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 16:37:19.353296 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:19.353256 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:19.377941 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:19.375817 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:37:19.603805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:19.575828 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:19.603805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:19.576408 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:37:19.670319 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:19.667932 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:37:19.670319 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:19.667990 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" podUID="3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 16:37:19.764935 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:19.762401 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:19.764935 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:19.762833 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:37:20.281484 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:20.281446 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:37:20.290069 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:20.290046 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:37:20.291847 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:20.291809 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:37:20.293664 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:20.293268 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:37:20.333816 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:20.320572 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:37:20.333816 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:20.320630 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 16:37:20.380952 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:37:20.380910 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:37:20.640680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:20.640597 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:20.669748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:20.664656 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:37:20.669748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:20.664713 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" podUID="3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 16:37:21.307074 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:21.307038 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 16:37:21.307334 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:21.307305 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 16:37:21.643195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:21.643118 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:37:21.643195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:21.643169 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" podUID="3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 16:37:22.437875 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:22.437841 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:22.645141 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:22.645103 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 16:37:22.645713 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:22.645673 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" podUID="3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 16:37:22.682157 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:22.682118 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:23.307453 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:23.307416 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:23.307628 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:23.307475 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:24.240468 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:24.240436 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:24.241038 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:24.240923 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:24.242072 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:24.242048 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 16:37:24.643744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:24.643632 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:24.643744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:24.643689 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" podUID="3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:24.765134 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:24.765094 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:24.927218 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:24.927131 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:25.440411 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:25.440382 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:25.441180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:25.441159 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:25.441921 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:25.441902 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:37:26.704748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:26.704699 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:28.597996 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:28.597968 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:29.377539 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.377500 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq"] Oct 30 16:37:29.382585 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.382563 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.422676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.422647 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq"] Oct 30 16:37:29.525530 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.525505 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.525700 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.525557 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.525821 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.525760 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.525821 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.525797 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.525943 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.525853 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.525943 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.525889 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j28mj\" (UniqueName: \"kubernetes.io/projected/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-kube-api-access-j28mj\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.526037 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.525965 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.526121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.526099 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.526181 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.526142 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.627325 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.627298 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.627829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.627359 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.627829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.627375 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.627829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.627400 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.627829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.627422 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j28mj\" (UniqueName: \"kubernetes.io/projected/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-kube-api-access-j28mj\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.627829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.627460 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.627829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.627550 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.627829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.627593 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.627829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.627708 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-workload-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.627829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.627797 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.628494 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.627945 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-credential-socket\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.628494 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.627965 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-workload-certs\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.628494 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.628150 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-data\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.628494 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.628365 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istiod-ca-cert\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.629944 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.629920 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-envoy\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.630263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.630246 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-podinfo\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.651700 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.651671 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-token\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.651877 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.651862 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j28mj\" (UniqueName: \"kubernetes.io/projected/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-kube-api-access-j28mj\") pod \"e2e-worker-w0-f3503b-openshift-default-668846555f-294lq\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:29.699907 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:29.699880 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:30.023164 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:30.023137 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq"] Oct 30 16:37:30.024201 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:37:30.024174 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22008ff_d02b_47ba_bc61_eb8302e0cd3d.slice/crio-26b6e6a5baf31b4b8bddd710cec25511fefa91bfd8895c61963f3db5efcf7237 WatchSource:0}: Error finding container 26b6e6a5baf31b4b8bddd710cec25511fefa91bfd8895c61963f3db5efcf7237: Status 404 returned error can't find the container with id 26b6e6a5baf31b4b8bddd710cec25511fefa91bfd8895c61963f3db5efcf7237 Oct 30 16:37:30.027174 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:30.027141 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:37:30.027272 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:30.027206 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:37:30.027272 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:30.027236 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:37:30.459801 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:30.451408 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:30.492574 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:30.492533 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" event={"ID":"a22008ff-d02b-47ba-bc61-eb8302e0cd3d","Type":"ContainerStarted","Data":"2d582dc6d79eed83186c8c59e14fd1f1587f9dd53d8d35895b17c98b98c07ccc"} Oct 30 16:37:30.492946 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:30.492912 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" event={"ID":"a22008ff-d02b-47ba-bc61-eb8302e0cd3d","Type":"ContainerStarted","Data":"26b6e6a5baf31b4b8bddd710cec25511fefa91bfd8895c61963f3db5efcf7237"} Oct 30 16:37:30.543968 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:30.543915 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" podStartSLOduration=1.543899012 podStartE2EDuration="1.543899012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:30.542844086 +0000 UTC m=+880.885518127" watchObservedRunningTime="2026-10-30 16:37:30.543899012 +0000 UTC m=+880.886573050" Oct 30 16:37:30.700775 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:30.700716 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:31.709330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:31.709301 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:37:31.710118 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:31.710085 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" podUID="a22008ff-d02b-47ba-bc61-eb8302e0cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 16:37:32.705224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:32.705182 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:37:32.705535 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:32.705498 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" podUID="a22008ff-d02b-47ba-bc61-eb8302e0cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 16:37:32.948591 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:32.945775 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w"] Oct 30 16:37:32.954007 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:32.953977 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:32.964907 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:32.964832 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w"] Oct 30 16:37:32.965182 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:32.965151 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:33.084756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.061185 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.093655 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.093620 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad8806d-d9cd-42f7-8507-a2f765165f43-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.097998 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.097968 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffqth\" (UniqueName: \"kubernetes.io/projected/5ad8806d-d9cd-42f7-8507-a2f765165f43-kube-api-access-ffqth\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.098185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.098166 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.103750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.098396 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.103936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.103912 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.104058 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.104042 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.104230 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.104214 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.109757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.104399 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.258865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.214480 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.258865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.214535 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad8806d-d9cd-42f7-8507-a2f765165f43-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.258865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.214595 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffqth\" (UniqueName: \"kubernetes.io/projected/5ad8806d-d9cd-42f7-8507-a2f765165f43-kube-api-access-ffqth\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.258865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.214640 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.258865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.214775 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.258865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.214800 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.258865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.214825 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.258865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.214893 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.258865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.214962 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.258865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.215560 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.258865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.215850 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.258865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.216510 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad8806d-d9cd-42f7-8507-a2f765165f43-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.258865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.220425 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.258865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.220679 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.258865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.256216 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.380749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.316828 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:33.380749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.330833 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.380749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.343642 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 16:37:33.380749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.344077 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 16:37:33.380749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.344354 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 16:37:33.507749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.450471 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffqth\" (UniqueName: \"kubernetes.io/projected/5ad8806d-d9cd-42f7-8507-a2f765165f43-kube-api-access-ffqth\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.547804 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.547528 4861 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:33.665516 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.665481 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:33.704346 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.704303 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:37:33.704678 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.704618 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" podUID="a22008ff-d02b-47ba-bc61-eb8302e0cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 16:37:33.971816 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.959280 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:34.031751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:33.989802 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:37:34.031751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.027922 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:34.242933 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.236444 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:37:34.317974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.317930 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:34.344838 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.344803 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:37:34.365012 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.364962 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 16:37:34.366465 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.366417 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 16:37:34.367286 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.367254 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 16:37:34.396052 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.396023 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:37:34.396332 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.396303 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:37:34.396537 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.396516 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:37:34.622802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.620235 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:37:34.721850 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.721652 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:37:34.730775 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.730745 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:37:34.744977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.744935 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" podUID="a22008ff-d02b-47ba-bc61-eb8302e0cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 16:37:34.770340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.770305 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:37:34.812638 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.812570 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:37:34.878945 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:34.878651 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:37:35.179582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:35.177659 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w"] Oct 30 16:37:35.180000 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:37:35.179962 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad8806d_d9cd_42f7_8507_a2f765165f43.slice/crio-b7d0d5503901b1ec04d606a2c3d43b253d1a4dd0c1f2e7d462515a57229ecf69 WatchSource:0}: Error finding container b7d0d5503901b1ec04d606a2c3d43b253d1a4dd0c1f2e7d462515a57229ecf69: Status 404 returned error can't find the container with id b7d0d5503901b1ec04d606a2c3d43b253d1a4dd0c1f2e7d462515a57229ecf69 Oct 30 16:37:35.183817 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:35.183778 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:37:35.184011 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:35.183987 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:37:35.184180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:35.184161 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:37:35.700280 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:35.700251 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:37:35.700458 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:35.700302 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" podUID="a22008ff-d02b-47ba-bc61-eb8302e0cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 16:37:35.853783 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:35.850204 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:35.879979 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:35.879942 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" event={"ID":"5ad8806d-d9cd-42f7-8507-a2f765165f43","Type":"ContainerStarted","Data":"a8c4a891042fe5e514c770c893232a33aff4964c193e3126a72cdd12215a0fab"} Oct 30 16:37:35.880144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:35.880007 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" event={"ID":"5ad8806d-d9cd-42f7-8507-a2f765165f43","Type":"ContainerStarted","Data":"b7d0d5503901b1ec04d606a2c3d43b253d1a4dd0c1f2e7d462515a57229ecf69"} Oct 30 16:37:35.949827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:35.949079 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" podStartSLOduration=3.94905709 podStartE2EDuration="3.94905709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:35.937103385 +0000 UTC m=+886.279777424" watchObservedRunningTime="2026-10-30 16:37:35.94905709 +0000 UTC m=+886.291731129" Oct 30 16:37:35.984637 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:35.984607 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:36.253676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:36.253555 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:36.700647 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:36.700570 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:37:36.700647 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:36.700623 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" podUID="a22008ff-d02b-47ba-bc61-eb8302e0cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 16:37:36.985844 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:36.985812 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:37:36.986028 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:36.985864 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" podUID="5ad8806d-d9cd-42f7-8507-a2f765165f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 16:37:37.720753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:37.700904 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 16:37:37.720753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:37.700968 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" podUID="a22008ff-d02b-47ba-bc61-eb8302e0cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 16:37:37.984895 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:37.984805 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:37:37.984895 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:37.984862 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" podUID="5ad8806d-d9cd-42f7-8507-a2f765165f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 16:37:38.440844 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:38.439486 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:38.984970 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:38.984937 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:37:38.985468 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:38.985120 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" podUID="5ad8806d-d9cd-42f7-8507-a2f765165f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 16:37:39.701758 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:39.700830 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:39.701758 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:39.700901 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" podUID="a22008ff-d02b-47ba-bc61-eb8302e0cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:39.985248 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:39.985216 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:37:39.985643 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:39.985279 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" podUID="5ad8806d-d9cd-42f7-8507-a2f765165f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 16:37:40.499042 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:40.499004 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:40.657757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:40.656801 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:40.660757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:40.658235 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:40.660757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:40.659210 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:37:40.984629 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:40.984600 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:37:40.984807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:40.984667 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" podUID="5ad8806d-d9cd-42f7-8507-a2f765165f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 16:37:41.867779 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:41.859319 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:41.986748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:41.986512 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:37:41.986748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:41.986587 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" podUID="5ad8806d-d9cd-42f7-8507-a2f765165f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 16:37:43.510748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:43.503989 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:43.546747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:43.545185 4861 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:43.630760 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:43.602445 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5"] Oct 30 16:37:43.897385 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:43.897280 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5"] Oct 30 16:37:43.897966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:43.897833 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:43.921476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:43.921453 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:43.923794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:43.923777 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:43.947111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:43.943755 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:37:44.006840 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.006795 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.007011 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.006899 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvxtg\" (UniqueName: \"kubernetes.io/projected/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-kube-api-access-mvxtg\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.007011 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.006940 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.007011 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.006977 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.007212 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.007083 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.007212 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.007136 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.007212 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.007165 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.007374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.007241 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.007374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.007291 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.109082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.109053 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.109226 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.109213 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.109289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.109271 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.109348 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.109302 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.109401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.109383 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.109454 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.109433 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.109923 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.109873 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.109923 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.109893 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.110158 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.110140 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.110337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.110309 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvxtg\" (UniqueName: \"kubernetes.io/projected/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-kube-api-access-mvxtg\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.110463 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.110358 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.110463 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.110156 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.110677 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.110647 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.110827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.110663 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.112253 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.112233 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.113009 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.112987 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.145167 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.145133 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.145872 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.145845 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvxtg\" (UniqueName: \"kubernetes.io/projected/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-kube-api-access-mvxtg\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.251152 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.251127 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:44.578893 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.578862 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5"] Oct 30 16:37:44.579801 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:37:44.579776 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceeed7ea_7144_48e0_8d06_b0a10f26f50a.slice/crio-249f35e5694bef1304168bac35533ad250ed94b6535658bbabeec7de314e7aed WatchSource:0}: Error finding container 249f35e5694bef1304168bac35533ad250ed94b6535658bbabeec7de314e7aed: Status 404 returned error can't find the container with id 249f35e5694bef1304168bac35533ad250ed94b6535658bbabeec7de314e7aed Oct 30 16:37:44.582337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.582303 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:37:44.582448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.582366 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:37:44.582448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:44.582394 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:37:45.059545 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:45.059500 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" event={"ID":"ceeed7ea-7144-48e0-8d06-b0a10f26f50a","Type":"ContainerStarted","Data":"792f3c5a18ddce125f0d562a1ccab9b7a2c8444f19f4b078102ad2458f45b48a"} Oct 30 16:37:45.059788 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:45.059595 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" event={"ID":"ceeed7ea-7144-48e0-8d06-b0a10f26f50a","Type":"ContainerStarted","Data":"249f35e5694bef1304168bac35533ad250ed94b6535658bbabeec7de314e7aed"} Oct 30 16:37:45.251541 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:45.251506 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:45.253149 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:45.253122 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:37:45.253274 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:45.253172 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" podUID="ceeed7ea-7144-48e0-8d06-b0a10f26f50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 16:37:45.487936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:45.487584 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:46.252485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:46.252452 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:37:46.252911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:46.252502 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" podUID="ceeed7ea-7144-48e0-8d06-b0a10f26f50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 16:37:47.251999 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:47.251966 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:37:47.252174 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:47.252029 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" podUID="ceeed7ea-7144-48e0-8d06-b0a10f26f50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 16:37:47.327488 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:47.327459 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:48.252031 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:48.251995 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:37:48.252211 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:48.252050 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" podUID="ceeed7ea-7144-48e0-8d06-b0a10f26f50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 16:37:49.252063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:49.252027 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:37:49.252484 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:49.252088 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" podUID="ceeed7ea-7144-48e0-8d06-b0a10f26f50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 16:37:49.297834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:49.297781 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:49.933245 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:49.933219 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:37:49.933433 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:49.933419 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:37:50.252038 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:50.252010 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:37:50.252215 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:50.252057 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" podUID="ceeed7ea-7144-48e0-8d06-b0a10f26f50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 16:37:50.292202 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:37:50.292165 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:37:51.193634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:51.193586 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:51.665747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:51.665067 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:51.679748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:51.666533 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:51.692791 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:51.684475 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:37:51.732748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:51.728334 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" podStartSLOduration=8.728312245 podStartE2EDuration="8.728312245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:45.127648016 +0000 UTC m=+895.470322056" watchObservedRunningTime="2026-10-30 16:37:51.728312245 +0000 UTC m=+902.070986284" Oct 30 16:37:52.891755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:52.884809 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:53.549663 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:53.549618 4861 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Oct 30 16:37:53.665134 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:53.665050 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 16:37:53.731019 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:53.730964 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:53.774917 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:53.774861 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:53.815978 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:53.815880 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:53.885898 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:53.885854 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:53.900754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:53.897072 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:37:54.606599 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:54.606567 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:37:54.712110 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:54.712064 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 16:37:54.716752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:54.713085 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:37:54.775864 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:54.775819 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:54.794424 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:54.786902 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:37:55.074757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:54.823810 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 16:37:55.074757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:55.017516 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:55.074757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:55.018288 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" podUID="6abf00fa-1784-428a-9ddc-afa6963b2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:55.104124 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:55.104094 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:37:55.402059 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:55.401971 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:37:55.560809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:55.560782 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:37:55.565745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:55.561649 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:37:55.774848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:55.774805 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:57.612351 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:57.612318 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:58.601264 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:58.601230 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:37:59.565665 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:37:59.565622 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:01.382750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:01.376967 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:03.221692 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:03.221635 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:05.695751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:05.692292 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:07.786890 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:07.786851 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:09.661903 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:09.661870 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:10.759848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:10.759815 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:11.497868 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:11.497836 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:13.321946 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:13.321904 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:15.142063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:15.138494 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:16.897349 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:16.897312 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:18.048240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.048208 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p"] Oct 30 16:38:18.053906 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.053886 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.071653 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.071627 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2"] Oct 30 16:38:18.077408 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.077383 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.089054 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.089027 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p"] Oct 30 16:38:18.101844 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.101559 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2"] Oct 30 16:38:18.114112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114083 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.114253 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114123 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262816dd-5373-4ee6-8624-727fd8b649c8-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.114253 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114159 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262816dd-5373-4ee6-8624-727fd8b649c8-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.114253 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114186 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.114253 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114220 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.114253 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114234 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.114456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114258 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj9c8\" (UniqueName: \"kubernetes.io/projected/b9db1487-cac4-489d-b0fb-e63b12a1703b-kube-api-access-rj9c8\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.114456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114302 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.114456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114321 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.114456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114349 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-482tf\" (UniqueName: \"kubernetes.io/projected/262816dd-5373-4ee6-8624-727fd8b649c8-kube-api-access-482tf\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.114456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114384 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.114456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114430 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.114456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114453 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9db1487-cac4-489d-b0fb-e63b12a1703b-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.114694 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114498 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.114694 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114525 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.114694 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114563 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.114694 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114576 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262816dd-5373-4ee6-8624-727fd8b649c8-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.114694 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.114590 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.215396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.215368 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.215595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.215406 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262816dd-5373-4ee6-8624-727fd8b649c8-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.215595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.215443 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262816dd-5373-4ee6-8624-727fd8b649c8-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.215595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.215486 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.215595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.215537 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.215595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.215560 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.215959 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.215606 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj9c8\" (UniqueName: \"kubernetes.io/projected/b9db1487-cac4-489d-b0fb-e63b12a1703b-kube-api-access-rj9c8\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.215959 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.215661 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.215959 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.215694 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.215959 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.215773 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-482tf\" (UniqueName: \"kubernetes.io/projected/262816dd-5373-4ee6-8624-727fd8b649c8-kube-api-access-482tf\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.215959 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.215892 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.216221 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.215972 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.216221 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.215988 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.216221 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.216016 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9db1487-cac4-489d-b0fb-e63b12a1703b-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.216221 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.216036 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.216221 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.216063 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.216221 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.216103 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.216221 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.216124 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.216221 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.216124 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.216221 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.216163 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.216221 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.216188 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262816dd-5373-4ee6-8624-727fd8b649c8-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.216221 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.216211 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.216785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.216382 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.216785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.216613 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.216785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.216650 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.216982 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.216961 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.217319 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.217295 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262816dd-5373-4ee6-8624-727fd8b649c8-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.217611 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.217587 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9db1487-cac4-489d-b0fb-e63b12a1703b-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.218539 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.218511 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.218844 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.218827 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262816dd-5373-4ee6-8624-727fd8b649c8-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.218994 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.218971 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.219185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.219171 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.239651 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.239621 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262816dd-5373-4ee6-8624-727fd8b649c8-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.239936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.239911 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.240015 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.239944 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-482tf\" (UniqueName: \"kubernetes.io/projected/262816dd-5373-4ee6-8624-727fd8b649c8-kube-api-access-482tf\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.240135 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.240118 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj9c8\" (UniqueName: \"kubernetes.io/projected/b9db1487-cac4-489d-b0fb-e63b12a1703b-kube-api-access-rj9c8\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.367651 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.367588 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:18.403446 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.403414 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:18.697263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.697235 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p"] Oct 30 16:38:18.698174 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:38:18.698147 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9db1487_cac4_489d_b0fb_e63b12a1703b.slice/crio-b4d1fcb1368491306ee502bd7a9cbeb6504426b37aa5b59be6efe75b65fcd1c0 WatchSource:0}: Error finding container b4d1fcb1368491306ee502bd7a9cbeb6504426b37aa5b59be6efe75b65fcd1c0: Status 404 returned error can't find the container with id b4d1fcb1368491306ee502bd7a9cbeb6504426b37aa5b59be6efe75b65fcd1c0 Oct 30 16:38:18.700550 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.700519 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:38:18.700637 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.700582 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:38:18.700637 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.700607 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:38:18.741170 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.741146 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2"] Oct 30 16:38:18.753900 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:38:18.753879 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262816dd_5373_4ee6_8624_727fd8b649c8.slice/crio-ca15d1db81cac60498df72bd402e88e24a28765443502fab957b959002ac6c54 WatchSource:0}: Error finding container ca15d1db81cac60498df72bd402e88e24a28765443502fab957b959002ac6c54: Status 404 returned error can't find the container with id ca15d1db81cac60498df72bd402e88e24a28765443502fab957b959002ac6c54 Oct 30 16:38:18.756425 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.756395 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:38:18.756481 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.756455 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:38:18.756525 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.756483 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:38:18.889341 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:18.889311 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:19.001921 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:19.001879 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" event={"ID":"262816dd-5373-4ee6-8624-727fd8b649c8","Type":"ContainerStarted","Data":"db432519444c6d0b3cfe88986ad1bd600d42373e9802bf7e552e33d426cb2a3d"} Oct 30 16:38:19.002447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:19.002269 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" event={"ID":"262816dd-5373-4ee6-8624-727fd8b649c8","Type":"ContainerStarted","Data":"ca15d1db81cac60498df72bd402e88e24a28765443502fab957b959002ac6c54"} Oct 30 16:38:19.006353 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:19.006322 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" event={"ID":"b9db1487-cac4-489d-b0fb-e63b12a1703b","Type":"ContainerStarted","Data":"0c53813042be3d532dc9097d09324cb8d7d811697ef5b163863be946af2795d7"} Oct 30 16:38:19.007632 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:19.007604 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" event={"ID":"b9db1487-cac4-489d-b0fb-e63b12a1703b","Type":"ContainerStarted","Data":"b4d1fcb1368491306ee502bd7a9cbeb6504426b37aa5b59be6efe75b65fcd1c0"} Oct 30 16:38:19.052092 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:19.052035 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" podStartSLOduration=1.052017156 podStartE2EDuration="1.052017156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:19.050128363 +0000 UTC m=+929.392802401" watchObservedRunningTime="2026-10-30 16:38:19.052017156 +0000 UTC m=+929.394691194" Oct 30 16:38:19.098206 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:19.098136 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" podStartSLOduration=1.098115137 podStartE2EDuration="1.098115137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:19.095943329 +0000 UTC m=+929.438617369" watchObservedRunningTime="2026-10-30 16:38:19.098115137 +0000 UTC m=+929.440789177" Oct 30 16:38:19.367987 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:19.367910 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:19.369688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:19.369661 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:38:19.369827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:19.369712 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:38:19.404004 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:19.403973 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:19.405180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:19.405152 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:38:19.405295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:19.405207 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 16:38:20.290358 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:38:20.290325 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:38:20.368344 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:20.368308 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:38:20.368537 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:20.368372 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:38:20.424265 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:20.424230 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:38:20.424765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:20.424714 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 16:38:20.856750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:20.856234 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:21.386752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:21.370291 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:38:21.386752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:21.370375 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:38:21.411811 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:21.407772 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:38:21.411811 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:21.407837 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 16:38:22.368191 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:22.368153 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:38:22.368476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:22.368446 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:38:22.405191 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:22.405152 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:38:22.405858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:22.405825 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 16:38:22.569439 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:22.569395 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 16:38:22.613351 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:22.613305 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:22.614161 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:22.614129 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:22.848750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:22.847821 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:22.888789 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:22.885797 4861 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded" Oct 30 16:38:23.368149 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:23.368099 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:38:23.368435 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:23.368405 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:38:23.406870 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:23.406822 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:23.410395 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:23.410369 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:38:23.410603 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:23.410580 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 16:38:23.539928 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:23.539869 4861 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:23.571898 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:23.571857 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:23.573266 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:23.573243 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:38:23.617097 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:23.617050 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:23.617746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:23.615701 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:23.619070 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:23.617934 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:38:23.632116 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:23.632092 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:38:24.066757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:24.048135 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:38:24.488077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:24.070555 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:38:24.488077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:24.418917 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 16:38:24.488077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:24.425434 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:38:24.488077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:24.437482 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:38:24.488077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:24.437533 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 16:38:24.488077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:24.437616 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:38:24.488077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:24.437641 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:38:24.576952 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:24.576907 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:24.664751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:24.648437 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:24.664751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:24.648920 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:24.916851 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:24.915918 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:24.916851 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:24.916018 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" podUID="6abf00fa-1784-428a-9ddc-afa6963b2985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.093961 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:25.093903 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:38:25.372811 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:25.363826 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:25.372811 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:25.363924 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.372811 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:25.368874 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:38:25.372811 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:25.368947 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:38:25.426765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:25.415641 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:38:25.426765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:25.415700 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 16:38:25.426765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:25.419824 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.581749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:25.577845 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.749551 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:25.749508 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.785792 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:25.762755 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.911803 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:25.909382 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:38:26.376787 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:26.367954 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:38:26.376787 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:26.368011 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:38:26.413710 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:26.413285 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:38:26.413710 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:26.413353 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 16:38:26.421839 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:26.421804 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.541203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:26.540837 4861 patch_prober.go:29] interesting pod/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:26.541203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:26.540917 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" podUID="3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.608780 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:26.598552 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.794863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:26.794814 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.800852 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:26.800814 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.833208 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:26.833164 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.188314 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:27.186184 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:38:27.382843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:27.374180 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:38:27.382843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:27.374258 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:38:27.408752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:27.405087 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:38:27.412845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:27.412803 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 16:38:27.422945 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:27.422893 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.586912 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:27.586877 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:27.587205 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:27.587175 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" podUID="12ea7c70-5553-4086-b4f5-dc5895c4f0a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.613191 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:27.613152 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.745181 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:27.744843 4861 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.917041 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:27.913829 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 16:38:27.935294 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:27.935067 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:38:27.941910 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:27.939803 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:38:28.189044 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:28.183980 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:38:28.231751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:28.227871 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:38:28.286627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:28.286540 4861 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:28.290484 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:28.289486 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:38:28.292488 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:28.292470 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:38:28.356795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:28.356764 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:28.369142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:28.368178 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:38:28.369431 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:28.369385 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:38:28.460687 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:28.460316 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:38:28.460687 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:28.460396 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 16:38:28.892145 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:28.892115 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:29.039635 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:29.039587 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:29.223896 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:29.223867 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:29.368105 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:29.368074 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:38:29.368304 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:29.368128 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:38:29.424268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:29.424236 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:38:29.424530 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:29.424500 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 16:38:30.181402 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:30.181371 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:30.353116 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:30.353081 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:30.368647 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:30.368576 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 16:38:30.368810 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:30.368636 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 16:38:30.403825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:30.403795 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:38:30.403998 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:30.403845 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 16:38:30.493904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:30.493870 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:30.644855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:30.644769 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:31.400217 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:31.400189 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:31.400744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:31.400506 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:31.401690 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:31.401662 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:38:31.408824 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:31.408804 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:31.409075 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:31.409053 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:31.409911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:31.409894 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:38:32.510945 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:32.510909 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:34.437257 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:34.437122 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:35.876204 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:35.876174 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:36.020817 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:36.020783 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:36.208530 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:36.208449 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:39.216770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:39.206208 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:38:39.443866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:39.443814 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.590757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:39.588829 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.652893 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:39.652841 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 16:38:39.653979 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:39.653157 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.854816 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:39.851391 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:40.246807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:40.225599 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:38:40.408739 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:40.408234 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:40.408739 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:40.408311 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.504833 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:40.477217 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.593763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:40.591147 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:38:40.593763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:40.591225 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:38:40.593763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:40.592152 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.679748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:40.638176 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:38:40.679748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:40.656563 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.679748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:40.657773 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:40.679748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:40.659116 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:38:40.679748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:40.659183 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:38:41.558957 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:41.558903 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:41.595079 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:41.595037 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:41.645879 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:41.645833 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:41.685820 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:41.659609 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:41.743290 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:41.659643 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 16:38:42.229764 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:42.229677 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:42.340431 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:42.340400 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:38:42.400590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:42.400418 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:38:42.484141 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:42.484056 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:38:42.504314 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:42.504285 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:38:42.590533 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:42.590496 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:42.596053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:42.596010 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:42.692092 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:42.692062 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:38:43.497190 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:43.497159 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:45.395055 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:45.395021 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:47.202545 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:47.202510 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:49.003490 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:49.003364 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:50.235640 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:50.235588 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:50.291952 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:38:50.291914 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:38:51.022978 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:51.022900 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:52.795598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:52.795561 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:54.594229 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:54.594190 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:56.448795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:56.448748 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:38:58.333696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:38:58.326266 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:00.191331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:00.191303 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:01.868704 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:01.868670 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:02.502745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:02.502462 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:03.349751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:03.348879 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.413865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:03.413813 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 16:39:03.509749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:03.508634 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.551942 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:03.551879 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.553545 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:03.552469 4861 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:03.667830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:03.666820 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:04.286745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:04.256242 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:04.355750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:04.350814 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:04.395750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:04.379227 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:39:04.448780 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:04.442977 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:04.448780 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:04.443950 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:39:04.577054 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:04.576931 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:04.578712 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:04.578687 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:39:04.604749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:04.602773 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 16:39:04.619001 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:04.618971 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:39:04.824568 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:04.824523 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:04.825927 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:04.825901 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:39:04.913802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:04.863606 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:05.357859 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:05.357814 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 16:39:05.451200 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:05.451158 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:05.604879 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:05.604827 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 16:39:05.607531 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:05.607488 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:05.906751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:05.826317 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:06.367875 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:06.367821 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 16:39:06.370406 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:06.369854 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:06.436774 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:06.435815 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" podUID="5a3cf99a-1f00-4e00-891c-01b06d04a4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:06.460420 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:06.460379 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:06.609863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:06.609818 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 16:39:06.630149 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:06.619780 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 16:39:06.788444 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:06.788072 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:06.788444 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:06.788135 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" podUID="866b3779-6bd0-4c61-a9cf-96161b1dc6d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:06.829748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:06.828783 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:39:07.002746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:06.997109 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:39:07.002746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:06.998776 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:39:07.046754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:07.046258 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:39:07.382810 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:07.375157 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 16:39:07.628327 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:07.628296 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:39:07.779276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:07.779238 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:07.781300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:07.781262 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" podUID="ceeed7ea-7144-48e0-8d06-b0a10f26f50a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:07.953777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:07.952849 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x7592 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:07.953777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:07.952907 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" podUID="5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:08.453452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:08.453403 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 16:39:08.781176 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:08.781061 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:39:09.018753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:09.016050 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-smq4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:09.018753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:09.016139 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-smq4m" podUID="f67a6d47-cf04-4307-af5b-ecb10e0612b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:09.289703 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:09.288781 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:09.670911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:09.670681 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-cfqjd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:09.672191 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:09.671202 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" podUID="c2cb80d7-4995-4584-8781-0c5921d6115c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:10.864960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:10.864909 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:11.032549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:11.032516 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:11.322673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:11.322643 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:12.632084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:12.632053 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:12.687458 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:12.687430 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9"] Oct 30 16:39:12.691567 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:12.691538 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" Oct 30 16:39:12.725582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:12.725536 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8295b2\"" Oct 30 16:39:12.731774 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:12.731744 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9"] Oct 30 16:39:12.750556 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:12.750518 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79"] Oct 30 16:39:12.755999 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:12.755976 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" Oct 30 16:39:12.779084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:12.779062 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79"] Oct 30 16:39:12.779525 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:12.779506 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8295b2-dockercfg-pcf9c\"" Oct 30 16:39:12.782900 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:12.782873 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:12.927802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:12.913967 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8vzf\" (UniqueName: \"kubernetes.io/projected/bc8a280e-2bfb-41e2-90ad-f92c9832e30a-kube-api-access-g8vzf\") pod \"payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9\" (UID: \"bc8a280e-2bfb-41e2-90ad-f92c9832e30a\") " pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" Oct 30 16:39:12.927802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:12.914043 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8a280e-2bfb-41e2-90ad-f92c9832e30a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9\" (UID: \"bc8a280e-2bfb-41e2-90ad-f92c9832e30a\") " pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" Oct 30 16:39:12.927802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:12.914126 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda3b43f-1c7d-44f5-85ef-e34b9dcd1431-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79\" (UID: \"fda3b43f-1c7d-44f5-85ef-e34b9dcd1431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" Oct 30 16:39:12.927802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:12.914155 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwj87\" (UniqueName: \"kubernetes.io/projected/fda3b43f-1c7d-44f5-85ef-e34b9dcd1431-kube-api-access-jwj87\") pod \"payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79\" (UID: \"fda3b43f-1c7d-44f5-85ef-e34b9dcd1431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" Oct 30 16:39:13.030452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:13.030390 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8vzf\" (UniqueName: \"kubernetes.io/projected/bc8a280e-2bfb-41e2-90ad-f92c9832e30a-kube-api-access-g8vzf\") pod \"payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9\" (UID: \"bc8a280e-2bfb-41e2-90ad-f92c9832e30a\") " pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" Oct 30 16:39:13.030777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:13.030749 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8a280e-2bfb-41e2-90ad-f92c9832e30a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9\" (UID: \"bc8a280e-2bfb-41e2-90ad-f92c9832e30a\") " pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" Oct 30 16:39:13.030991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:13.030966 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda3b43f-1c7d-44f5-85ef-e34b9dcd1431-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79\" (UID: \"fda3b43f-1c7d-44f5-85ef-e34b9dcd1431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" Oct 30 16:39:13.031119 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:13.031098 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwj87\" (UniqueName: \"kubernetes.io/projected/fda3b43f-1c7d-44f5-85ef-e34b9dcd1431-kube-api-access-jwj87\") pod \"payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79\" (UID: \"fda3b43f-1c7d-44f5-85ef-e34b9dcd1431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" Oct 30 16:39:13.033186 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:13.033147 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8a280e-2bfb-41e2-90ad-f92c9832e30a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9\" (UID: \"bc8a280e-2bfb-41e2-90ad-f92c9832e30a\") " pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" Oct 30 16:39:13.037972 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:13.034588 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda3b43f-1c7d-44f5-85ef-e34b9dcd1431-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79\" (UID: \"fda3b43f-1c7d-44f5-85ef-e34b9dcd1431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" Oct 30 16:39:13.105716 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:13.104357 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwj87\" (UniqueName: \"kubernetes.io/projected/fda3b43f-1c7d-44f5-85ef-e34b9dcd1431-kube-api-access-jwj87\") pod \"payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79\" (UID: \"fda3b43f-1c7d-44f5-85ef-e34b9dcd1431\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" Oct 30 16:39:13.105716 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:13.104358 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8vzf\" (UniqueName: \"kubernetes.io/projected/bc8a280e-2bfb-41e2-90ad-f92c9832e30a-kube-api-access-g8vzf\") pod \"payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9\" (UID: \"bc8a280e-2bfb-41e2-90ad-f92c9832e30a\") " pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" Oct 30 16:39:13.305997 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:13.305969 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" Oct 30 16:39:13.372311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:13.372073 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" Oct 30 16:39:13.585891 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:13.557612 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:13.641996 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:13.641965 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9"] Oct 30 16:39:13.644156 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:39:13.644127 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8a280e_2bfb_41e2_90ad_f92c9832e30a.slice/crio-c7eb3bcca60a4ba8103fe31703cbdd446c6ad690eccba914ba301f3357245d19 WatchSource:0}: Error finding container c7eb3bcca60a4ba8103fe31703cbdd446c6ad690eccba914ba301f3357245d19: Status 404 returned error can't find the container with id c7eb3bcca60a4ba8103fe31703cbdd446c6ad690eccba914ba301f3357245d19 Oct 30 16:39:13.704524 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:39:13.704484 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda3b43f_1c7d_44f5_85ef_e34b9dcd1431.slice/crio-fb9af3d798044a5e064d18bd0f8bd5c4b7de2bcf378392106df078a40aca486d WatchSource:0}: Error finding container fb9af3d798044a5e064d18bd0f8bd5c4b7de2bcf378392106df078a40aca486d: Status 404 returned error can't find the container with id fb9af3d798044a5e064d18bd0f8bd5c4b7de2bcf378392106df078a40aca486d Oct 30 16:39:13.705134 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:13.705043 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79"] Oct 30 16:39:13.713659 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:13.713631 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" event={"ID":"bc8a280e-2bfb-41e2-90ad-f92c9832e30a","Type":"ContainerStarted","Data":"c7eb3bcca60a4ba8103fe31703cbdd446c6ad690eccba914ba301f3357245d19"} Oct 30 16:39:13.715040 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:13.715015 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" event={"ID":"fda3b43f-1c7d-44f5-85ef-e34b9dcd1431","Type":"ContainerStarted","Data":"fb9af3d798044a5e064d18bd0f8bd5c4b7de2bcf378392106df078a40aca486d"} Oct 30 16:39:14.579500 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:14.579472 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:14.723806 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:14.723773 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" event={"ID":"bc8a280e-2bfb-41e2-90ad-f92c9832e30a","Type":"ContainerStarted","Data":"871718777fc6b2c493161e23f49e16ae4e8340e88ed07648099d83148c6fd459"} Oct 30 16:39:14.724243 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:14.723835 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" Oct 30 16:39:14.725538 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:14.725511 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" event={"ID":"fda3b43f-1c7d-44f5-85ef-e34b9dcd1431","Type":"ContainerStarted","Data":"ddc38eb51050f69292126e71655c93e83779d53c9750fd708564206016958f2f"} Oct 30 16:39:14.725674 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:14.725566 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" Oct 30 16:39:14.758933 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:14.758905 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:14.768448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:14.768407 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" podStartSLOduration=2.768395268 podStartE2EDuration="2.768395268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:14.765093685 +0000 UTC m=+985.107767723" watchObservedRunningTime="2026-10-30 16:39:14.768395268 +0000 UTC m=+985.111069329" Oct 30 16:39:14.808196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:14.808145 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" podStartSLOduration=2.808132876 podStartE2EDuration="2.808132876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:14.807402605 +0000 UTC m=+985.150076646" watchObservedRunningTime="2026-10-30 16:39:14.808132876 +0000 UTC m=+985.150806914" Oct 30 16:39:14.935652 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:14.935570 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg"] Oct 30 16:39:14.944186 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:14.944159 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:14.965237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:14.965057 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-82381960-openshift-default-dockercfg-hffwd\"" Oct 30 16:39:14.974566 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:14.974534 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg"] Oct 30 16:39:15.025782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.025717 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:15.076833 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.076804 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngkg2\" (UniqueName: \"kubernetes.io/projected/38ac300a-fa56-4c84-8c54-ff4d677e9555-kube-api-access-ngkg2\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.076986 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.076851 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-envoy\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.076986 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.076879 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-token\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.077075 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.077009 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-workload-socket\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.077075 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.077051 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-data\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.077146 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.077102 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-workload-certs\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.077181 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.077144 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-podinfo\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.077285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.077271 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-credential-socket\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.077330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.077296 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ac300a-fa56-4c84-8c54-ff4d677e9555-istiod-ca-cert\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.178347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.178316 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-workload-socket\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.178545 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.178374 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-data\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.178610 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.178576 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-workload-certs\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.178877 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.178854 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-data\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.179004 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.178856 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-workload-socket\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.179004 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.178925 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-workload-certs\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.179106 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.179086 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-podinfo\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.179279 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.179261 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-credential-socket\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.179395 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.179292 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ac300a-fa56-4c84-8c54-ff4d677e9555-istiod-ca-cert\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.179395 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.179353 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngkg2\" (UniqueName: \"kubernetes.io/projected/38ac300a-fa56-4c84-8c54-ff4d677e9555-kube-api-access-ngkg2\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.179512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.179404 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-envoy\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.179512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.179440 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-token\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.179627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.179589 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-credential-socket\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.180491 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.180471 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ac300a-fa56-4c84-8c54-ff4d677e9555-istiod-ca-cert\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.183217 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.183190 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-podinfo\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.189012 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.188955 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-envoy\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.199202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.199174 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:15.216711 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.216680 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngkg2\" (UniqueName: \"kubernetes.io/projected/38ac300a-fa56-4c84-8c54-ff4d677e9555-kube-api-access-ngkg2\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.219540 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.219508 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-token\") pod \"e2e-mt-82381960-openshift-default-68d7558b65-5pgqg\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.261350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.261323 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:15.376989 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.376955 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:15.557751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.551914 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:15.681563 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.680706 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg"] Oct 30 16:39:15.693432 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:39:15.693400 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ac300a_fa56_4c84_8c54_ff4d677e9555.slice/crio-aa9cfc520ce88f534834b68f60083b5eb4a7a7a77bd1ed2d2b120db146cb3697 WatchSource:0}: Error finding container aa9cfc520ce88f534834b68f60083b5eb4a7a7a77bd1ed2d2b120db146cb3697: Status 404 returned error can't find the container with id aa9cfc520ce88f534834b68f60083b5eb4a7a7a77bd1ed2d2b120db146cb3697 Oct 30 16:39:15.696414 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.696347 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:39:15.696521 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.696459 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:39:15.696521 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.696501 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:39:15.732479 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:15.732449 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" event={"ID":"38ac300a-fa56-4c84-8c54-ff4d677e9555","Type":"ContainerStarted","Data":"aa9cfc520ce88f534834b68f60083b5eb4a7a7a77bd1ed2d2b120db146cb3697"} Oct 30 16:39:16.740929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:16.740886 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" event={"ID":"38ac300a-fa56-4c84-8c54-ff4d677e9555","Type":"ContainerStarted","Data":"5edecec21ad47c959bc92a5fefbf9345b9a03c059e1120f80f486ca42406d37d"} Oct 30 16:39:16.814479 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:16.814426 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" podStartSLOduration=2.814409924 podStartE2EDuration="2.814409924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:16.799981367 +0000 UTC m=+987.142655407" watchObservedRunningTime="2026-10-30 16:39:16.814409924 +0000 UTC m=+987.157083963" Oct 30 16:39:17.262234 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:17.262206 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:17.268452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:17.268427 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:17.397256 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:17.397229 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:17.747443 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:17.747409 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:17.748366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:17.748345 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:39:18.731264 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:18.731225 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:18.871754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:18.871692 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:19.299748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:19.299706 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:19.445434 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:19.445399 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:20.290049 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:39:20.290006 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:39:20.497020 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:20.496955 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:20.580589 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:20.580516 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7"] Oct 30 16:39:20.586554 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:20.586533 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" Oct 30 16:39:20.603588 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:20.603560 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7"] Oct 30 16:39:20.609876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:20.609849 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c40832c-edc2-4075-be6f-22547bd79a82-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7\" (UID: \"5c40832c-edc2-4075-be6f-22547bd79a82\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" Oct 30 16:39:20.610001 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:20.609970 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdlzn\" (UniqueName: \"kubernetes.io/projected/5c40832c-edc2-4075-be6f-22547bd79a82-kube-api-access-mdlzn\") pod \"payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7\" (UID: \"5c40832c-edc2-4075-be6f-22547bd79a82\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" Oct 30 16:39:20.613986 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:20.613949 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0b09f1\"" Oct 30 16:39:20.640423 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:20.640397 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:20.809770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:20.809287 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c40832c-edc2-4075-be6f-22547bd79a82-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7\" (UID: \"5c40832c-edc2-4075-be6f-22547bd79a82\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" Oct 30 16:39:20.809770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:20.809404 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdlzn\" (UniqueName: \"kubernetes.io/projected/5c40832c-edc2-4075-be6f-22547bd79a82-kube-api-access-mdlzn\") pod \"payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7\" (UID: \"5c40832c-edc2-4075-be6f-22547bd79a82\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" Oct 30 16:39:20.894805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:20.893634 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdlzn\" (UniqueName: \"kubernetes.io/projected/5c40832c-edc2-4075-be6f-22547bd79a82-kube-api-access-mdlzn\") pod \"payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7\" (UID: \"5c40832c-edc2-4075-be6f-22547bd79a82\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" Oct 30 16:39:20.922751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:20.915706 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c40832c-edc2-4075-be6f-22547bd79a82-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7\" (UID: \"5c40832c-edc2-4075-be6f-22547bd79a82\") " pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" Oct 30 16:39:21.201907 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:21.201832 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" Oct 30 16:39:21.519329 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:21.519300 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7"] Oct 30 16:39:21.519879 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:39:21.519857 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c40832c_edc2_4075_be6f_22547bd79a82.slice/crio-dab6fbebea75d22950e93d38d99c9a167fa47b22103e5ccd95e3e6333a3cead6 WatchSource:0}: Error finding container dab6fbebea75d22950e93d38d99c9a167fa47b22103e5ccd95e3e6333a3cead6: Status 404 returned error can't find the container with id dab6fbebea75d22950e93d38d99c9a167fa47b22103e5ccd95e3e6333a3cead6 Oct 30 16:39:21.656507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:21.656475 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:21.926791 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:21.926695 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" event={"ID":"5c40832c-edc2-4075-be6f-22547bd79a82","Type":"ContainerStarted","Data":"95ed41af8e4ce15bd5650de6970a3f7d43bb8e5343e1042fe7adece440da3045"} Oct 30 16:39:21.926791 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:21.926763 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" event={"ID":"5c40832c-edc2-4075-be6f-22547bd79a82","Type":"ContainerStarted","Data":"dab6fbebea75d22950e93d38d99c9a167fa47b22103e5ccd95e3e6333a3cead6"} Oct 30 16:39:21.926990 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:21.926794 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" Oct 30 16:39:21.968435 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:21.968390 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" podStartSLOduration=1.968376913 podStartE2EDuration="1.968376913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:21.967982445 +0000 UTC m=+992.310656483" watchObservedRunningTime="2026-10-30 16:39:21.968376913 +0000 UTC m=+992.311050951" Oct 30 16:39:22.380332 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.380295 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:22.481090 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.481059 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq"] Oct 30 16:39:22.485482 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.485463 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" Oct 30 16:39:22.536117 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.536085 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d1af4a-8f20-4200-85b7-615c9475d573-plugins-config-volume\") pod \"payload-processing-e2e-mt-82381960-579cc97966-6k7zq\" (UID: \"50d1af4a-8f20-4200-85b7-615c9475d573\") " pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" Oct 30 16:39:22.536436 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.536255 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nhz4\" (UniqueName: \"kubernetes.io/projected/50d1af4a-8f20-4200-85b7-615c9475d573-kube-api-access-7nhz4\") pod \"payload-processing-e2e-mt-82381960-579cc97966-6k7zq\" (UID: \"50d1af4a-8f20-4200-85b7-615c9475d573\") " pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" Oct 30 16:39:22.544407 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.544382 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-82381960\"" Oct 30 16:39:22.548525 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.548504 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq"] Oct 30 16:39:22.556581 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.556551 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-82381960-dockercfg-l727f\"" Oct 30 16:39:22.571909 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.571883 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s"] Oct 30 16:39:22.577114 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.577088 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" Oct 30 16:39:22.601367 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.601342 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s"] Oct 30 16:39:22.637332 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.637252 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nhz4\" (UniqueName: \"kubernetes.io/projected/50d1af4a-8f20-4200-85b7-615c9475d573-kube-api-access-7nhz4\") pod \"payload-processing-e2e-mt-82381960-579cc97966-6k7zq\" (UID: \"50d1af4a-8f20-4200-85b7-615c9475d573\") " pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" Oct 30 16:39:22.637332 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.637302 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57333841-13b5-466b-8fc5-bf4c0b5ea649-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s\" (UID: \"57333841-13b5-466b-8fc5-bf4c0b5ea649\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" Oct 30 16:39:22.637519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.637385 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5fwl\" (UniqueName: \"kubernetes.io/projected/57333841-13b5-466b-8fc5-bf4c0b5ea649-kube-api-access-x5fwl\") pod \"payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s\" (UID: \"57333841-13b5-466b-8fc5-bf4c0b5ea649\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" Oct 30 16:39:22.637586 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.637519 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d1af4a-8f20-4200-85b7-615c9475d573-plugins-config-volume\") pod \"payload-processing-e2e-mt-82381960-579cc97966-6k7zq\" (UID: \"50d1af4a-8f20-4200-85b7-615c9475d573\") " pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" Oct 30 16:39:22.638295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.638273 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d1af4a-8f20-4200-85b7-615c9475d573-plugins-config-volume\") pod \"payload-processing-e2e-mt-82381960-579cc97966-6k7zq\" (UID: \"50d1af4a-8f20-4200-85b7-615c9475d573\") " pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" Oct 30 16:39:22.642597 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.642576 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:22.722202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.720017 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nhz4\" (UniqueName: \"kubernetes.io/projected/50d1af4a-8f20-4200-85b7-615c9475d573-kube-api-access-7nhz4\") pod \"payload-processing-e2e-mt-82381960-579cc97966-6k7zq\" (UID: \"50d1af4a-8f20-4200-85b7-615c9475d573\") " pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" Oct 30 16:39:22.746759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.740775 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5fwl\" (UniqueName: \"kubernetes.io/projected/57333841-13b5-466b-8fc5-bf4c0b5ea649-kube-api-access-x5fwl\") pod \"payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s\" (UID: \"57333841-13b5-466b-8fc5-bf4c0b5ea649\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" Oct 30 16:39:22.746759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.741059 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57333841-13b5-466b-8fc5-bf4c0b5ea649-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s\" (UID: \"57333841-13b5-466b-8fc5-bf4c0b5ea649\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" Oct 30 16:39:22.771742 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.768131 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57333841-13b5-466b-8fc5-bf4c0b5ea649-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s\" (UID: \"57333841-13b5-466b-8fc5-bf4c0b5ea649\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" Oct 30 16:39:22.788582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.788499 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5fwl\" (UniqueName: \"kubernetes.io/projected/57333841-13b5-466b-8fc5-bf4c0b5ea649-kube-api-access-x5fwl\") pod \"payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s\" (UID: \"57333841-13b5-466b-8fc5-bf4c0b5ea649\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" Oct 30 16:39:22.810302 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.810028 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" Oct 30 16:39:22.899576 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:22.899282 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" Oct 30 16:39:23.245839 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:23.245812 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq"] Oct 30 16:39:23.246459 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:39:23.246434 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d1af4a_8f20_4200_85b7_615c9475d573.slice/crio-91472f2d800994a198fdf253b723e867c5fa01ebf6ba628aa766f26c4babe118 WatchSource:0}: Error finding container 91472f2d800994a198fdf253b723e867c5fa01ebf6ba628aa766f26c4babe118: Status 404 returned error can't find the container with id 91472f2d800994a198fdf253b723e867c5fa01ebf6ba628aa766f26c4babe118 Oct 30 16:39:23.330019 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:23.329990 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s"] Oct 30 16:39:24.009299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:24.009255 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" event={"ID":"50d1af4a-8f20-4200-85b7-615c9475d573","Type":"ContainerStarted","Data":"c375ccc76ffe115d26f321dacd7500c0c4f8eccb5f621f6fda43fd6837287524"} Oct 30 16:39:24.009958 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:24.009325 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" event={"ID":"50d1af4a-8f20-4200-85b7-615c9475d573","Type":"ContainerStarted","Data":"91472f2d800994a198fdf253b723e867c5fa01ebf6ba628aa766f26c4babe118"} Oct 30 16:39:24.009958 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:24.009480 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" Oct 30 16:39:24.011780 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:24.011754 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" event={"ID":"57333841-13b5-466b-8fc5-bf4c0b5ea649","Type":"ContainerStarted","Data":"3552dbb28daa4e46d3c8dcde4eb8c572563a253df3a8308721968bc21f53460e"} Oct 30 16:39:24.012008 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:24.011801 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" event={"ID":"57333841-13b5-466b-8fc5-bf4c0b5ea649","Type":"ContainerStarted","Data":"f7e84a86e03ac895d0b674bc9e76e790a6dec609ec9ee2a5bf538fef4d9036e3"} Oct 30 16:39:24.012008 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:24.011872 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" Oct 30 16:39:24.070178 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:24.070126 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" podStartSLOduration=2.070112182 podStartE2EDuration="2.070112182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:24.061110372 +0000 UTC m=+994.403784416" watchObservedRunningTime="2026-10-30 16:39:24.070112182 +0000 UTC m=+994.412786221" Oct 30 16:39:24.106113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:24.106049 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" podStartSLOduration=2.106034897 podStartE2EDuration="2.106034897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:24.103098047 +0000 UTC m=+994.445772088" watchObservedRunningTime="2026-10-30 16:39:24.106034897 +0000 UTC m=+994.448708935" Oct 30 16:39:24.155453 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:24.155412 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:24.334099 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:24.332284 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:25.760753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:25.757136 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" Oct 30 16:39:25.760753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:25.759565 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" Oct 30 16:39:26.678528 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:26.678494 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:27.753306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.753273 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg"] Oct 30 16:39:27.758471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.758447 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.786305 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.786277 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.786549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.786502 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.786671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.786567 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.786671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.786592 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.786671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.786630 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.786849 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.786698 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.786849 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.786748 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfrnb\" (UniqueName: \"kubernetes.io/projected/64a4e246-be6d-4d34-92d2-7d69ff5fb454-kube-api-access-jfrnb\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.786947 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.786878 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.786947 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.786922 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.795083 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.795057 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg"] Oct 30 16:39:27.888155 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.888126 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.888325 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.888176 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.888325 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.888298 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.888397 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.888357 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.888477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.888458 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.888518 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.888494 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfrnb\" (UniqueName: \"kubernetes.io/projected/64a4e246-be6d-4d34-92d2-7d69ff5fb454-kube-api-access-jfrnb\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.888882 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.888579 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.888882 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.888613 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.888882 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.888761 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.888882 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.888818 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.889179 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.889073 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.889179 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.889103 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.889179 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.889157 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.889397 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.889380 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.890924 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.890898 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.891274 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.891252 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.911985 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.911954 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:27.912158 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:27.912138 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfrnb\" (UniqueName: \"kubernetes.io/projected/64a4e246-be6d-4d34-92d2-7d69ff5fb454-kube-api-access-jfrnb\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:28.073448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:28.073365 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:28.402387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:28.402360 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg"] Oct 30 16:39:28.402910 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:39:28.402845 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a4e246_be6d_4d34_92d2_7d69ff5fb454.slice/crio-cc62673fcf3c20ff86ebc7ac8ec9b72b50fcd0ec54d15b1d9bd1312c1ba16b2c WatchSource:0}: Error finding container cc62673fcf3c20ff86ebc7ac8ec9b72b50fcd0ec54d15b1d9bd1312c1ba16b2c: Status 404 returned error can't find the container with id cc62673fcf3c20ff86ebc7ac8ec9b72b50fcd0ec54d15b1d9bd1312c1ba16b2c Oct 30 16:39:28.405475 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:28.405442 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:39:28.405556 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:28.405505 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:39:28.405556 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:28.405533 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:39:28.500450 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:28.500422 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:29.197237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:29.197199 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" event={"ID":"64a4e246-be6d-4d34-92d2-7d69ff5fb454","Type":"ContainerStarted","Data":"ce2e704956cb7dc579d9f67211585ff7a08c3ee14895ee00bde3742939142747"} Oct 30 16:39:29.197764 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:29.197261 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" event={"ID":"64a4e246-be6d-4d34-92d2-7d69ff5fb454","Type":"ContainerStarted","Data":"cc62673fcf3c20ff86ebc7ac8ec9b72b50fcd0ec54d15b1d9bd1312c1ba16b2c"} Oct 30 16:39:30.074221 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:30.074186 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:30.076401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:30.076369 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 16:39:30.076542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:30.076428 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" podUID="64a4e246-be6d-4d34-92d2-7d69ff5fb454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 16:39:30.405706 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:30.405607 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:31.074339 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:31.074307 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 16:39:31.074520 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:31.074365 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" podUID="64a4e246-be6d-4d34-92d2-7d69ff5fb454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 16:39:32.074391 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.074356 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 16:39:32.074796 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.074423 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" podUID="64a4e246-be6d-4d34-92d2-7d69ff5fb454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 16:39:32.227435 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.227365 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" podStartSLOduration=5.227345552 podStartE2EDuration="5.227345552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:29.284370063 +0000 UTC m=+999.627044100" watchObservedRunningTime="2026-10-30 16:39:32.227345552 +0000 UTC m=+1002.570019592" Oct 30 16:39:32.231574 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.231549 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5"] Oct 30 16:39:32.236119 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.236099 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:32.253515 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.253490 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5"] Oct 30 16:39:32.263176 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.262329 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0b09f1-serving-cert\"" Oct 30 16:39:32.283427 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.283399 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-maas-api-tls\") pod \"maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:32.283549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.283444 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-openshift-service-ca\") pod \"maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:32.283549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.283469 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpvmh\" (UniqueName: \"kubernetes.io/projected/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-kube-api-access-cpvmh\") pod \"maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:32.283675 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.283656 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-metrics-certs\") pod \"maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:32.330042 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.329958 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:32.385614 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.385584 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-maas-api-tls\") pod \"maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:32.385812 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.385623 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-openshift-service-ca\") pod \"maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:32.385812 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.385643 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpvmh\" (UniqueName: \"kubernetes.io/projected/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-kube-api-access-cpvmh\") pod \"maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:32.385812 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.385760 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-metrics-certs\") pod \"maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:32.385955 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:39:32.385760 4861 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-serving-cert: secret "maas-api-e2e-api-a-0b09f1-serving-cert" not found Oct 30 16:39:32.385955 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:39:32.385952 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-maas-api-tls podName:5bdd45e6-87c4-4168-9bb3-9a627cbf5813 nodeName:}" failed. No retries permitted until 2026-10-30 16:39:32.885930117 +0000 UTC m=+1003.228604134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-maas-api-tls") pod "maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" (UID: "5bdd45e6-87c4-4168-9bb3-9a627cbf5813") : secret "maas-api-e2e-api-a-0b09f1-serving-cert" not found Oct 30 16:39:32.386318 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.386294 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-openshift-service-ca\") pod \"maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:32.388494 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.388468 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-metrics-certs\") pod \"maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:32.454746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.443924 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpvmh\" (UniqueName: \"kubernetes.io/projected/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-kube-api-access-cpvmh\") pod \"maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:32.476789 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.476241 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:32.639914 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.639796 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:32.929839 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.929756 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-maas-api-tls\") pod \"maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:32.932235 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:32.932208 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-maas-api-tls\") pod \"maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:33.024747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:33.023890 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" Oct 30 16:39:33.074129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:33.074096 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 16:39:33.074285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:33.074149 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" podUID="64a4e246-be6d-4d34-92d2-7d69ff5fb454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 16:39:33.151819 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:33.151784 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:33.851026 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:33.843206 4861 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:39:33.875617 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:33.875591 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5"] Oct 30 16:39:34.074698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:34.074659 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 16:39:34.074906 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:34.074765 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" podUID="64a4e246-be6d-4d34-92d2-7d69ff5fb454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 16:39:34.267543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:34.267514 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" event={"ID":"5bdd45e6-87c4-4168-9bb3-9a627cbf5813","Type":"ContainerStarted","Data":"7f67d853f443569e59427900376f0345e932765fdd7eae3a70c9f4fec79529ce"} Oct 30 16:39:34.573429 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:34.573336 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:35.063162 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:35.063131 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" Oct 30 16:39:35.063360 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:35.063218 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" Oct 30 16:39:35.074177 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:35.074152 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 16:39:35.074318 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:35.074213 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" podUID="64a4e246-be6d-4d34-92d2-7d69ff5fb454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 16:39:35.283519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:35.283481 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" event={"ID":"5bdd45e6-87c4-4168-9bb3-9a627cbf5813","Type":"ContainerStarted","Data":"5dbe3cc5b61bd43ffa717c68fbf502747969dafe11e9b153b5b955abdb5901ee"} Oct 30 16:39:35.284117 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:35.283546 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:35.329807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:35.329690 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" podStartSLOduration=2.927028312 podStartE2EDuration="3.329675543s" totalImagesPullingTime="402.647231ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:35.327231046 +0000 UTC m=+1005.669905100" watchObservedRunningTime="2026-10-30 16:39:35.329675543 +0000 UTC m=+1005.672349581" Oct 30 16:39:36.077987 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:36.077959 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:36.078474 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:36.078457 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:36.079151 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:36.079132 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:39:36.559674 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:36.559627 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:36.704752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:36.704561 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:36.965082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:36.965006 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:38.745104 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:38.745058 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:40.600241 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:40.600209 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:41.297798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:41.297771 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:39:42.404444 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.404417 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:42.552663 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.552447 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:42.793584 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.793520 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:42.805255 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.805232 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw"] Oct 30 16:39:42.809933 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.809915 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.842243 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.842216 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw"] Oct 30 16:39:42.881037 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.881003 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpg9t\" (UniqueName: \"kubernetes.io/projected/8c6122cd-f566-44be-b180-ae3a2578ddfa-kube-api-access-cpg9t\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.881227 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.881083 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.881227 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.881178 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.881353 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.881239 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.881353 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.881344 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6122cd-f566-44be-b180-ae3a2578ddfa-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.881462 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.881414 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.881567 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.881552 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.881629 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.881580 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.881682 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.881663 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.935886 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.935855 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:42.983470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.983441 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpg9t\" (UniqueName: \"kubernetes.io/projected/8c6122cd-f566-44be-b180-ae3a2578ddfa-kube-api-access-cpg9t\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.983635 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.983493 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.983635 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.983532 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.983635 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.983567 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.983635 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.983606 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6122cd-f566-44be-b180-ae3a2578ddfa-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.983893 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.983646 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.983893 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.983717 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.983893 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.983764 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.983893 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.983804 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.984155 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.983950 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.984155 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.984116 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.984292 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.984182 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.984292 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.984244 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.984370 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.984350 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6122cd-f566-44be-b180-ae3a2578ddfa-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.986225 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.986199 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:42.986473 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:42.986452 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:43.007955 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:43.007931 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpg9t\" (UniqueName: \"kubernetes.io/projected/8c6122cd-f566-44be-b180-ae3a2578ddfa-kube-api-access-cpg9t\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:43.008080 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:43.008060 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:43.123908 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:43.123833 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:43.453372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:43.453300 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw"] Oct 30 16:39:43.457254 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:39:43.457231 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6122cd_f566_44be_b180_ae3a2578ddfa.slice/crio-e9994032e266c456b470347fc63bd2784f935bead9861ae7b628b706bdb4d88d WatchSource:0}: Error finding container e9994032e266c456b470347fc63bd2784f935bead9861ae7b628b706bdb4d88d: Status 404 returned error can't find the container with id e9994032e266c456b470347fc63bd2784f935bead9861ae7b628b706bdb4d88d Oct 30 16:39:43.459928 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:43.459895 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:39:43.460020 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:43.459969 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:39:43.460020 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:43.460012 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:39:44.377823 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:44.377704 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" event={"ID":"8c6122cd-f566-44be-b180-ae3a2578ddfa","Type":"ContainerStarted","Data":"0e28be9e0f031e1c608721b40b004044c2d700643ea82e1877bc4a68ae610aac"} Oct 30 16:39:44.377823 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:44.377797 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" event={"ID":"8c6122cd-f566-44be-b180-ae3a2578ddfa","Type":"ContainerStarted","Data":"e9994032e266c456b470347fc63bd2784f935bead9861ae7b628b706bdb4d88d"} Oct 30 16:39:44.498557 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:44.498522 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:45.125832 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:45.125786 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:45.126763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:45.126703 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 16:39:45.126888 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:45.126795 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" podUID="8c6122cd-f566-44be-b180-ae3a2578ddfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 16:39:46.126036 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:46.126000 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 16:39:46.126799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:46.126757 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" podUID="8c6122cd-f566-44be-b180-ae3a2578ddfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 16:39:46.252612 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:46.252577 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:46.677006 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:46.676974 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:46.906589 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:46.906558 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:47.125751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:47.124989 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 16:39:47.125751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:47.125079 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" podUID="8c6122cd-f566-44be-b180-ae3a2578ddfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 16:39:47.370549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:47.370515 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:47.633120 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:47.633065 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:48.125752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:48.124693 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 16:39:48.125752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:48.124785 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" podUID="8c6122cd-f566-44be-b180-ae3a2578ddfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 16:39:48.639743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:48.639024 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:49.126236 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:49.126200 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 16:39:49.126446 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:49.126297 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" podUID="8c6122cd-f566-44be-b180-ae3a2578ddfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 16:39:50.124177 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:50.124138 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 16:39:50.124656 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:50.124195 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" podUID="8c6122cd-f566-44be-b180-ae3a2578ddfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 16:39:50.290299 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:39:50.290257 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:39:50.380566 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:50.380469 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:50.591428 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:50.591398 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:51.080912 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:51.080874 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:51.236105 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:51.236077 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:51.236774 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:51.236752 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:51.237441 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:51.237424 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:39:51.245793 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:51.245768 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:51.301324 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:51.301280 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" podStartSLOduration=9.301268323 podStartE2EDuration="9.301268323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:44.474186396 +0000 UTC m=+1014.816860435" watchObservedRunningTime="2026-10-30 16:39:51.301268323 +0000 UTC m=+1021.643942361" Oct 30 16:39:53.141511 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.141455 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:53.274534 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.274492 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs"] Oct 30 16:39:53.280291 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.280270 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" Oct 30 16:39:53.295558 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.295529 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:53.300344 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.300325 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs"] Oct 30 16:39:53.322746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.322345 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-90df54-dockercfg-qq9l6\"" Oct 30 16:39:53.331751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.323044 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-90df54\"" Oct 30 16:39:53.391801 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.376390 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c86fa8-9c6e-4aac-92e4-434250b1b3a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs\" (UID: \"34c86fa8-9c6e-4aac-92e4-434250b1b3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" Oct 30 16:39:53.391801 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.376529 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jblvk\" (UniqueName: \"kubernetes.io/projected/34c86fa8-9c6e-4aac-92e4-434250b1b3a0-kube-api-access-jblvk\") pod \"payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs\" (UID: \"34c86fa8-9c6e-4aac-92e4-434250b1b3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" Oct 30 16:39:53.610747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.525623 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c86fa8-9c6e-4aac-92e4-434250b1b3a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs\" (UID: \"34c86fa8-9c6e-4aac-92e4-434250b1b3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" Oct 30 16:39:53.610747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.525783 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jblvk\" (UniqueName: \"kubernetes.io/projected/34c86fa8-9c6e-4aac-92e4-434250b1b3a0-kube-api-access-jblvk\") pod \"payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs\" (UID: \"34c86fa8-9c6e-4aac-92e4-434250b1b3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" Oct 30 16:39:53.610747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.542134 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c86fa8-9c6e-4aac-92e4-434250b1b3a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs\" (UID: \"34c86fa8-9c6e-4aac-92e4-434250b1b3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" Oct 30 16:39:53.610747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.605047 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:53.736811 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.648565 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jblvk\" (UniqueName: \"kubernetes.io/projected/34c86fa8-9c6e-4aac-92e4-434250b1b3a0-kube-api-access-jblvk\") pod \"payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs\" (UID: \"34c86fa8-9c6e-4aac-92e4-434250b1b3a0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" Oct 30 16:39:53.736811 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.648645 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng"] Oct 30 16:39:53.736811 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.689051 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" Oct 30 16:39:53.797237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.797203 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng"] Oct 30 16:39:53.834794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.828288 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44a2b96-5f28-4bee-89b1-3aa1967fdad8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng\" (UID: \"c44a2b96-5f28-4bee-89b1-3aa1967fdad8\") " pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" Oct 30 16:39:53.834794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.828629 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5h8j\" (UniqueName: \"kubernetes.io/projected/c44a2b96-5f28-4bee-89b1-3aa1967fdad8-kube-api-access-c5h8j\") pod \"payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng\" (UID: \"c44a2b96-5f28-4bee-89b1-3aa1967fdad8\") " pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" Oct 30 16:39:53.896818 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.895436 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" Oct 30 16:39:53.939701 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.932197 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44a2b96-5f28-4bee-89b1-3aa1967fdad8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng\" (UID: \"c44a2b96-5f28-4bee-89b1-3aa1967fdad8\") " pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" Oct 30 16:39:53.939701 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.932551 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5h8j\" (UniqueName: \"kubernetes.io/projected/c44a2b96-5f28-4bee-89b1-3aa1967fdad8-kube-api-access-c5h8j\") pod \"payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng\" (UID: \"c44a2b96-5f28-4bee-89b1-3aa1967fdad8\") " pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" Oct 30 16:39:53.939701 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.933972 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44a2b96-5f28-4bee-89b1-3aa1967fdad8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng\" (UID: \"c44a2b96-5f28-4bee-89b1-3aa1967fdad8\") " pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" Oct 30 16:39:53.961781 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:53.961710 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5h8j\" (UniqueName: \"kubernetes.io/projected/c44a2b96-5f28-4bee-89b1-3aa1967fdad8-kube-api-access-c5h8j\") pod \"payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng\" (UID: \"c44a2b96-5f28-4bee-89b1-3aa1967fdad8\") " pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" Oct 30 16:39:54.136880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:54.134898 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:54.136880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:54.135084 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" Oct 30 16:39:54.330855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:54.330819 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs"] Oct 30 16:39:54.593938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:54.593238 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng"] Oct 30 16:39:54.749505 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:54.749479 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:54.790553 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:54.790509 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" event={"ID":"c44a2b96-5f28-4bee-89b1-3aa1967fdad8","Type":"ContainerStarted","Data":"6befaf433708f8636586cac4edd7463b19c64346a7e8afb37274d0ef7cf9a230"} Oct 30 16:39:54.790698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:54.790604 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" event={"ID":"c44a2b96-5f28-4bee-89b1-3aa1967fdad8","Type":"ContainerStarted","Data":"ffc167143f14886a2c94fdfb8bc1b920b06ee74dbcadf070b626cbd1197ff910"} Oct 30 16:39:54.790698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:54.790635 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" Oct 30 16:39:54.792484 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:54.792462 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" event={"ID":"34c86fa8-9c6e-4aac-92e4-434250b1b3a0","Type":"ContainerStarted","Data":"fe5fea3d3a28d7b553df7b4213144c93cd0719dc7cbf1c9120d88de26bd16407"} Oct 30 16:39:54.792578 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:54.792507 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" Oct 30 16:39:54.792578 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:54.792525 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" event={"ID":"34c86fa8-9c6e-4aac-92e4-434250b1b3a0","Type":"ContainerStarted","Data":"ce6b73303f890a163624ea462989232819f72491744f7f8ccc5b324fcf5db2db"} Oct 30 16:39:54.835804 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:54.835752 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" podStartSLOduration=1.835709646 podStartE2EDuration="1.835709646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:54.835053468 +0000 UTC m=+1025.177727505" watchObservedRunningTime="2026-10-30 16:39:54.835709646 +0000 UTC m=+1025.178383685" Oct 30 16:39:54.880228 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:54.880124 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" podStartSLOduration=1.8801045520000002 podStartE2EDuration="1.880104552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:54.876159803 +0000 UTC m=+1025.218833843" watchObservedRunningTime="2026-10-30 16:39:54.880104552 +0000 UTC m=+1025.222778591" Oct 30 16:39:55.358447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:55.358377 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:55.530759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:55.526637 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:56.194478 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:56.192540 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:58.379460 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:58.379433 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:59.558126 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:59.558094 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:59.741018 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:59.740981 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:39:59.912311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:39:59.912208 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:00.686802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:00.685412 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:01.339692 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.339658 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:01.472124 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.472092 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p"] Oct 30 16:40:01.477713 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.477691 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" Oct 30 16:40:01.502104 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.502080 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-83d885-dockercfg-db4c2\"" Oct 30 16:40:01.502404 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.502377 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:01.514964 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.514939 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-83d885\"" Oct 30 16:40:01.521067 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.521041 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p"] Oct 30 16:40:01.607807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.603709 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcsh\" (UniqueName: \"kubernetes.io/projected/143c2049-fabb-4d96-8c30-6c11b24b2c4c-kube-api-access-mzcsh\") pod \"payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p\" (UID: \"143c2049-fabb-4d96-8c30-6c11b24b2c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" Oct 30 16:40:01.607807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.604014 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143c2049-fabb-4d96-8c30-6c11b24b2c4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p\" (UID: \"143c2049-fabb-4d96-8c30-6c11b24b2c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" Oct 30 16:40:01.651749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.646785 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d"] Oct 30 16:40:01.767751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.748603 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143c2049-fabb-4d96-8c30-6c11b24b2c4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p\" (UID: \"143c2049-fabb-4d96-8c30-6c11b24b2c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" Oct 30 16:40:01.767751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.748823 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcsh\" (UniqueName: \"kubernetes.io/projected/143c2049-fabb-4d96-8c30-6c11b24b2c4c-kube-api-access-mzcsh\") pod \"payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p\" (UID: \"143c2049-fabb-4d96-8c30-6c11b24b2c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" Oct 30 16:40:01.767751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.750058 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143c2049-fabb-4d96-8c30-6c11b24b2c4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p\" (UID: \"143c2049-fabb-4d96-8c30-6c11b24b2c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" Oct 30 16:40:01.900019 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.897562 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:01.906430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.906042 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d"] Oct 30 16:40:01.909899 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.909861 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" Oct 30 16:40:01.937411 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.935529 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs"] Oct 30 16:40:01.938374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.938343 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcsh\" (UniqueName: \"kubernetes.io/projected/143c2049-fabb-4d96-8c30-6c11b24b2c4c-kube-api-access-mzcsh\") pod \"payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p\" (UID: \"143c2049-fabb-4d96-8c30-6c11b24b2c4c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" Oct 30 16:40:01.942702 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.942679 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs"] Oct 30 16:40:01.942832 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.942754 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:01.962119 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.961967 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-82381960\"/\"openshift-service-ca.crt\"" Oct 30 16:40:01.962119 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.962035 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-82381960\"/\"default-dockercfg-9rp7d\"" Oct 30 16:40:01.962119 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.962091 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-82381960\"/\"kube-root-ca.crt\"" Oct 30 16:40:01.963401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:01.963383 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-82381960\"/\"e2e-lifecycle-model-82381960-kserve-self-signed-certs\"" Oct 30 16:40:02.094751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.093035 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" Oct 30 16:40:02.104851 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.104808 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ec445f-a902-4ed5-b917-a8460f8e4179-tls-certs\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.127929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.105180 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-model-cache\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.127929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.123903 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-tmp-dir\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.127929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.124011 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-home\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.127929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.124070 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-dshm\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.127929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.124227 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d599d5-d4b9-4708-ba37-beecd9e6cfbe-plugins-config-volume\") pod \"payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d\" (UID: \"29d599d5-d4b9-4708-ba37-beecd9e6cfbe\") " pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" Oct 30 16:40:02.127929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.124434 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhxsb\" (UniqueName: \"kubernetes.io/projected/60ec445f-a902-4ed5-b917-a8460f8e4179-kube-api-access-bhxsb\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.127929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.124652 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pntm\" (UniqueName: \"kubernetes.io/projected/29d599d5-d4b9-4708-ba37-beecd9e6cfbe-kube-api-access-2pntm\") pod \"payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d\" (UID: \"29d599d5-d4b9-4708-ba37-beecd9e6cfbe\") " pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" Oct 30 16:40:02.239994 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.239958 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-model-cache\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.240822 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.240794 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-tmp-dir\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.241065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.241047 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-home\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.241227 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.241203 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-dshm\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.241464 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.241445 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d599d5-d4b9-4708-ba37-beecd9e6cfbe-plugins-config-volume\") pod \"payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d\" (UID: \"29d599d5-d4b9-4708-ba37-beecd9e6cfbe\") " pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" Oct 30 16:40:02.241705 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.241685 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhxsb\" (UniqueName: \"kubernetes.io/projected/60ec445f-a902-4ed5-b917-a8460f8e4179-kube-api-access-bhxsb\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.241971 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.241950 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pntm\" (UniqueName: \"kubernetes.io/projected/29d599d5-d4b9-4708-ba37-beecd9e6cfbe-kube-api-access-2pntm\") pod \"payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d\" (UID: \"29d599d5-d4b9-4708-ba37-beecd9e6cfbe\") " pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" Oct 30 16:40:02.242167 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.242124 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ec445f-a902-4ed5-b917-a8460f8e4179-tls-certs\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.244580 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.244495 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d599d5-d4b9-4708-ba37-beecd9e6cfbe-plugins-config-volume\") pod \"payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d\" (UID: \"29d599d5-d4b9-4708-ba37-beecd9e6cfbe\") " pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" Oct 30 16:40:02.253285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.253260 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-tmp-dir\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.253422 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.240594 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-model-cache\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.253541 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.253523 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-home\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.263409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.263093 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-dshm\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.263409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.263363 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ec445f-a902-4ed5-b917-a8460f8e4179-tls-certs\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.268862 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.268013 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhxsb\" (UniqueName: \"kubernetes.io/projected/60ec445f-a902-4ed5-b917-a8460f8e4179-kube-api-access-bhxsb\") pod \"e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.278966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.277981 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pntm\" (UniqueName: \"kubernetes.io/projected/29d599d5-d4b9-4708-ba37-beecd9e6cfbe-kube-api-access-2pntm\") pod \"payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d\" (UID: \"29d599d5-d4b9-4708-ba37-beecd9e6cfbe\") " pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" Oct 30 16:40:02.283672 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.283635 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:02.529846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.529823 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p"] Oct 30 16:40:02.531841 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:40:02.531797 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143c2049_fabb_4d96_8c30_6c11b24b2c4c.slice/crio-800de2288e88275f59395f69f754c17eeb3aee2afe8709ee24ff14ca7ad9f411 WatchSource:0}: Error finding container 800de2288e88275f59395f69f754c17eeb3aee2afe8709ee24ff14ca7ad9f411: Status 404 returned error can't find the container with id 800de2288e88275f59395f69f754c17eeb3aee2afe8709ee24ff14ca7ad9f411 Oct 30 16:40:02.575674 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.575632 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" Oct 30 16:40:02.626961 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.625931 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs"] Oct 30 16:40:02.809619 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.809534 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:02.969334 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.969307 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d"] Oct 30 16:40:02.978506 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.978477 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" event={"ID":"60ec445f-a902-4ed5-b917-a8460f8e4179","Type":"ContainerStarted","Data":"6ca8590df2e3d3accd932186bb18d79ba813f83c3fc5ce9ad42eb8472b1bf5a9"} Oct 30 16:40:02.978640 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.978538 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" event={"ID":"60ec445f-a902-4ed5-b917-a8460f8e4179","Type":"ContainerStarted","Data":"6ec5b57f7205744f4d499c03450cca89ccadd051a49a0c5fa07c939b6cb4f349"} Oct 30 16:40:02.980591 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.980568 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" event={"ID":"143c2049-fabb-4d96-8c30-6c11b24b2c4c","Type":"ContainerStarted","Data":"9a4f1dd68dd0f457625add2d202c132570b82072f34cf6d3671b0992b75493ac"} Oct 30 16:40:02.980716 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.980625 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" event={"ID":"143c2049-fabb-4d96-8c30-6c11b24b2c4c","Type":"ContainerStarted","Data":"800de2288e88275f59395f69f754c17eeb3aee2afe8709ee24ff14ca7ad9f411"} Oct 30 16:40:02.980716 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:02.980661 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" Oct 30 16:40:03.024925 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:03.024880 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" podStartSLOduration=2.024866179 podStartE2EDuration="2.024866179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:03.020903173 +0000 UTC m=+1033.363577212" watchObservedRunningTime="2026-10-30 16:40:03.024866179 +0000 UTC m=+1033.367540259" Oct 30 16:40:03.065399 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:03.065319 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" podStartSLOduration=2.065306094 podStartE2EDuration="2.065306094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:03.063998448 +0000 UTC m=+1033.406672487" watchObservedRunningTime="2026-10-30 16:40:03.065306094 +0000 UTC m=+1033.407980132" Oct 30 16:40:03.290571 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:03.286885 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:03.573506 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:03.571693 4861 patch_prober.go:29] interesting pod/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": read tcp 10.132.0.2:54594->10.132.0.88:15021: read: connection reset by peer" start-of-body= Oct 30 16:40:03.573506 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:03.572270 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" podUID="38ac300a-fa56-4c84-8c54-ff4d677e9555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": read tcp 10.132.0.2:54594->10.132.0.88:15021: read: connection reset by peer" Oct 30 16:40:03.991151 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:03.991114 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" event={"ID":"29d599d5-d4b9-4708-ba37-beecd9e6cfbe","Type":"ContainerStarted","Data":"3a2622d9487c9b642d43051ef2cc61dc20ecc4c06f59ddf87738b709444f9880"} Oct 30 16:40:03.991609 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:03.991175 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" event={"ID":"29d599d5-d4b9-4708-ba37-beecd9e6cfbe","Type":"ContainerStarted","Data":"c6415d329e771495f4538b65d28875f1b605f93240a6a30694bf4bb960fe84a9"} Oct 30 16:40:03.991609 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:03.991201 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" Oct 30 16:40:03.992918 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:03.992897 4861 generic.go:356] "Generic (PLEG): container finished" podID="38ac300a-fa56-4c84-8c54-ff4d677e9555" containerID="5edecec21ad47c959bc92a5fefbf9345b9a03c059e1120f80f486ca42406d37d" exitCode=0 Oct 30 16:40:03.992984 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:03.992966 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" event={"ID":"38ac300a-fa56-4c84-8c54-ff4d677e9555","Type":"ContainerDied","Data":"5edecec21ad47c959bc92a5fefbf9345b9a03c059e1120f80f486ca42406d37d"} Oct 30 16:40:03.993616 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:03.993603 4861 scope.go:122] "RemoveContainer" containerID="5edecec21ad47c959bc92a5fefbf9345b9a03c059e1120f80f486ca42406d37d" Oct 30 16:40:03.994948 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:03.994924 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:40:03.995018 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:03.994986 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:40:03.995055 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:03.995018 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:40:04.041323 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:04.041276 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" podStartSLOduration=3.041261684 podStartE2EDuration="3.041261684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:04.037074024 +0000 UTC m=+1034.379748062" watchObservedRunningTime="2026-10-30 16:40:04.041261684 +0000 UTC m=+1034.383935721" Oct 30 16:40:04.722750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:04.715532 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:05.028852 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:05.028754 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" event={"ID":"38ac300a-fa56-4c84-8c54-ff4d677e9555","Type":"ContainerStarted","Data":"5961c9cfb09f31cacd458122e3cacecd72bdb5c0b3e5ca2f314eaa6acc15dbaa"} Oct 30 16:40:05.262115 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:05.262079 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:40:05.262281 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:05.262233 4861 patch_prober.go:29] interesting pod/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 16:40:05.262324 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:05.262277 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" podUID="38ac300a-fa56-4c84-8c54-ff4d677e9555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 16:40:05.858572 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:05.858541 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" Oct 30 16:40:05.858767 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:05.858624 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" Oct 30 16:40:06.261878 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:06.261844 4861 patch_prober.go:29] interesting pod/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 16:40:06.262271 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:06.261899 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" podUID="38ac300a-fa56-4c84-8c54-ff4d677e9555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 16:40:06.738680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:06.738641 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:07.261988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:07.261958 4861 patch_prober.go:29] interesting pod/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 16:40:07.262453 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:07.262427 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" podUID="38ac300a-fa56-4c84-8c54-ff4d677e9555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 16:40:08.261632 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:08.261599 4861 patch_prober.go:29] interesting pod/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 16:40:08.261823 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:08.261658 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" podUID="38ac300a-fa56-4c84-8c54-ff4d677e9555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 16:40:08.645797 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:08.644649 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:09.261869 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:09.261828 4861 patch_prober.go:29] interesting pod/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 16:40:09.262079 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:09.261902 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" podUID="38ac300a-fa56-4c84-8c54-ff4d677e9555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 16:40:10.266343 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:10.266314 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:40:10.266811 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:10.266776 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:40:10.267630 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:10.267612 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:40:10.557323 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:10.557233 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:10.725754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:10.725034 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:12.297431 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:12.294144 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:12.311234 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:12.311209 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:12.689061 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:12.688976 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:13.123098 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:13.123068 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:40:13.996477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:13.996449 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" Oct 30 16:40:14.700452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:14.700408 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:15.033244 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:15.033219 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" Oct 30 16:40:16.531419 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:16.531382 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:18.220154 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.220122 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv"] Oct 30 16:40:18.224854 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.224834 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:18.262390 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.262320 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-83d885-serving-cert\"" Oct 30 16:40:18.262390 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.262365 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv"] Oct 30 16:40:18.316086 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.316044 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgdz6\" (UniqueName: \"kubernetes.io/projected/51859589-1590-41e5-a1be-73680e4ced59-kube-api-access-zgdz6\") pod \"maas-api-e2e-api-b-83d885-86549d5df4-vqtgv\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:18.316246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.316137 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51859589-1590-41e5-a1be-73680e4ced59-openshift-service-ca\") pod \"maas-api-e2e-api-b-83d885-86549d5df4-vqtgv\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:18.316339 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.316186 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-metrics-certs\") pod \"maas-api-e2e-api-b-83d885-86549d5df4-vqtgv\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:18.316506 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.316488 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-maas-api-tls\") pod \"maas-api-e2e-api-b-83d885-86549d5df4-vqtgv\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:18.344089 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.344054 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:18.425586 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.424310 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgdz6\" (UniqueName: \"kubernetes.io/projected/51859589-1590-41e5-a1be-73680e4ced59-kube-api-access-zgdz6\") pod \"maas-api-e2e-api-b-83d885-86549d5df4-vqtgv\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:18.425586 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.424400 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51859589-1590-41e5-a1be-73680e4ced59-openshift-service-ca\") pod \"maas-api-e2e-api-b-83d885-86549d5df4-vqtgv\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:18.425586 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.424449 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-metrics-certs\") pod \"maas-api-e2e-api-b-83d885-86549d5df4-vqtgv\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:18.425586 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.424614 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-maas-api-tls\") pod \"maas-api-e2e-api-b-83d885-86549d5df4-vqtgv\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:18.425586 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:40:18.424911 4861 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-serving-cert: secret "maas-api-e2e-api-b-83d885-serving-cert" not found Oct 30 16:40:18.425586 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:40:18.424996 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-maas-api-tls podName:51859589-1590-41e5-a1be-73680e4ced59 nodeName:}" failed. No retries permitted until 2026-10-30 16:40:18.924973058 +0000 UTC m=+1049.267647091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-maas-api-tls") pod "maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" (UID: "51859589-1590-41e5-a1be-73680e4ced59") : secret "maas-api-e2e-api-b-83d885-serving-cert" not found Oct 30 16:40:18.425586 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.425510 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51859589-1590-41e5-a1be-73680e4ced59-openshift-service-ca\") pod \"maas-api-e2e-api-b-83d885-86549d5df4-vqtgv\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:18.475476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.475395 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-metrics-certs\") pod \"maas-api-e2e-api-b-83d885-86549d5df4-vqtgv\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:18.485762 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.481773 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgdz6\" (UniqueName: \"kubernetes.io/projected/51859589-1590-41e5-a1be-73680e4ced59-kube-api-access-zgdz6\") pod \"maas-api-e2e-api-b-83d885-86549d5df4-vqtgv\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:18.493775 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.490276 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:18.637130 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.637100 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:18.932448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.932360 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-maas-api-tls\") pod \"maas-api-e2e-api-b-83d885-86549d5df4-vqtgv\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:18.934978 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:18.934955 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-maas-api-tls\") pod \"maas-api-e2e-api-b-83d885-86549d5df4-vqtgv\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:19.139826 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:19.139788 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:19.526480 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:19.526194 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv"] Oct 30 16:40:20.173129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:20.173093 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" event={"ID":"51859589-1590-41e5-a1be-73680e4ced59","Type":"ContainerStarted","Data":"720c1a0fc81ebc57506cd25d818961fd935566e28d9ff35f41720f8f1169c168"} Oct 30 16:40:20.173299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:20.173144 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" event={"ID":"51859589-1590-41e5-a1be-73680e4ced59","Type":"ContainerStarted","Data":"4b56d318bb9260b796b588fcb497d90beb04c1793d2843ab4cbb00a77c152acc"} Oct 30 16:40:20.173363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:20.173303 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:20.218343 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:20.218284 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" podStartSLOduration=1.7839246370000001 podStartE2EDuration="2.218270332s" totalImagesPullingTime="434.345695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:20.21662468 +0000 UTC m=+1050.559298720" watchObservedRunningTime="2026-10-30 16:40:20.218270332 +0000 UTC m=+1050.560944369" Oct 30 16:40:20.291015 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:40:20.290935 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:40:20.383754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:20.383066 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:22.200353 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:22.200316 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:23.583850 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:23.583813 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:24.000125 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:24.000089 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:25.801169 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:25.801133 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:26.184873 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:26.184788 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:40:27.603977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:27.603946 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:29.520234 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:29.520199 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:30.293446 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:30.293413 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:31.299472 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:31.299434 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:31.626204 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:31.626128 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:32.097659 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:32.097616 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:32.266291 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:32.266260 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:33.432657 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:33.432629 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:35.965714 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:35.965678 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:36.148749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:36.147333 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:36.755214 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:36.755181 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:37.865167 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:37.865135 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:37.936337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:37.936302 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-x7592 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:40:37.936527 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:37.936422 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" podUID="5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:38.010685 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:38.010653 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:38.642115 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:38.642084 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:40.654208 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:40.654171 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:42.514684 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:42.514634 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:42.659911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:42.659876 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:44.493911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:44.493882 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:44.818022 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:44.817936 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:46.453212 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:46.453177 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:48.283514 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:48.283479 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:49.634665 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:49.634633 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9"] Oct 30 16:40:49.667741 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:49.667698 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79"] Oct 30 16:40:49.802565 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:49.802535 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:49.945293 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:49.945203 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:50.125385 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:50.125354 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:50.506922 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:40:50.506863 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:40:51.912759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:51.912705 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:53.673262 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:53.673227 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:54.635356 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:54.635286 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" podUID="bc8a280e-2bfb-41e2-90ad-f92c9832e30a" containerName="payload-processing" containerID="cri-o://871718777fc6b2c493161e23f49e16ae4e8340e88ed07648099d83148c6fd459" gracePeriod=25 Oct 30 16:40:54.667986 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:54.667933 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" podUID="fda3b43f-1c7d-44f5-85ef-e34b9dcd1431" containerName="payload-pre-processing" containerID="cri-o://ddc38eb51050f69292126e71655c93e83779d53c9750fd708564206016958f2f" gracePeriod=25 Oct 30 16:40:54.974092 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:54.974035 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" Oct 30 16:40:54.979979 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:54.979945 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc8a280e-2bfb-41e2-90ad-f92c9832e30a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8a280e-2bfb-41e2-90ad-f92c9832e30a-plugins-config-volume\") pod \"bc8a280e-2bfb-41e2-90ad-f92c9832e30a\" (UID: \"bc8a280e-2bfb-41e2-90ad-f92c9832e30a\") " Oct 30 16:40:54.980130 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:54.980113 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc8a280e-2bfb-41e2-90ad-f92c9832e30a-kube-api-access-g8vzf\" (UniqueName: \"kubernetes.io/projected/bc8a280e-2bfb-41e2-90ad-f92c9832e30a-kube-api-access-g8vzf\") pod \"bc8a280e-2bfb-41e2-90ad-f92c9832e30a\" (UID: \"bc8a280e-2bfb-41e2-90ad-f92c9832e30a\") " Oct 30 16:40:54.980314 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:54.980282 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc8a280e-2bfb-41e2-90ad-f92c9832e30a-plugins-config-volume" pod "bc8a280e-2bfb-41e2-90ad-f92c9832e30a" (UID: "bc8a280e-2bfb-41e2-90ad-f92c9832e30a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:40:54.980676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:54.980654 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc8a280e-2bfb-41e2-90ad-f92c9832e30a-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:40:54.982734 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:54.982692 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc8a280e-2bfb-41e2-90ad-f92c9832e30a-kube-api-access-g8vzf" pod "bc8a280e-2bfb-41e2-90ad-f92c9832e30a" (UID: "bc8a280e-2bfb-41e2-90ad-f92c9832e30a"). InnerVolumeSpecName "kube-api-access-g8vzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:55.006053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.006032 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" Oct 30 16:40:55.081365 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.081333 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda3b43f-1c7d-44f5-85ef-e34b9dcd1431-kube-api-access-jwj87\" (UniqueName: \"kubernetes.io/projected/fda3b43f-1c7d-44f5-85ef-e34b9dcd1431-kube-api-access-jwj87\") pod \"fda3b43f-1c7d-44f5-85ef-e34b9dcd1431\" (UID: \"fda3b43f-1c7d-44f5-85ef-e34b9dcd1431\") " Oct 30 16:40:55.081709 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.081664 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fda3b43f-1c7d-44f5-85ef-e34b9dcd1431-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda3b43f-1c7d-44f5-85ef-e34b9dcd1431-plugins-config-volume\") pod \"fda3b43f-1c7d-44f5-85ef-e34b9dcd1431\" (UID: \"fda3b43f-1c7d-44f5-85ef-e34b9dcd1431\") " Oct 30 16:40:55.082337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.082278 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda3b43f-1c7d-44f5-85ef-e34b9dcd1431-plugins-config-volume" pod "fda3b43f-1c7d-44f5-85ef-e34b9dcd1431" (UID: "fda3b43f-1c7d-44f5-85ef-e34b9dcd1431"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:40:55.082524 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.082474 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fda3b43f-1c7d-44f5-85ef-e34b9dcd1431-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:40:55.082524 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.082498 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8vzf\" (UniqueName: \"kubernetes.io/projected/bc8a280e-2bfb-41e2-90ad-f92c9832e30a-kube-api-access-g8vzf\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:40:55.084039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.084012 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda3b43f-1c7d-44f5-85ef-e34b9dcd1431-kube-api-access-jwj87" pod "fda3b43f-1c7d-44f5-85ef-e34b9dcd1431" (UID: "fda3b43f-1c7d-44f5-85ef-e34b9dcd1431"). InnerVolumeSpecName "kube-api-access-jwj87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:55.184694 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.184625 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwj87\" (UniqueName: \"kubernetes.io/projected/fda3b43f-1c7d-44f5-85ef-e34b9dcd1431-kube-api-access-jwj87\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:40:55.495777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.495741 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:55.663075 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.663046 4861 generic.go:356] "Generic (PLEG): container finished" podID="bc8a280e-2bfb-41e2-90ad-f92c9832e30a" containerID="871718777fc6b2c493161e23f49e16ae4e8340e88ed07648099d83148c6fd459" exitCode=0 Oct 30 16:40:55.663215 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.663099 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" Oct 30 16:40:55.663215 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.663133 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" event={"ID":"bc8a280e-2bfb-41e2-90ad-f92c9832e30a","Type":"ContainerDied","Data":"871718777fc6b2c493161e23f49e16ae4e8340e88ed07648099d83148c6fd459"} Oct 30 16:40:55.663348 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.663218 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9" event={"ID":"bc8a280e-2bfb-41e2-90ad-f92c9832e30a","Type":"ContainerDied","Data":"c7eb3bcca60a4ba8103fe31703cbdd446c6ad690eccba914ba301f3357245d19"} Oct 30 16:40:55.663348 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.663246 4861 scope.go:122] "RemoveContainer" containerID="871718777fc6b2c493161e23f49e16ae4e8340e88ed07648099d83148c6fd459" Oct 30 16:40:55.664759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.664737 4861 generic.go:356] "Generic (PLEG): container finished" podID="fda3b43f-1c7d-44f5-85ef-e34b9dcd1431" containerID="ddc38eb51050f69292126e71655c93e83779d53c9750fd708564206016958f2f" exitCode=0 Oct 30 16:40:55.664860 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.664801 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" Oct 30 16:40:55.664860 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.664840 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" event={"ID":"fda3b43f-1c7d-44f5-85ef-e34b9dcd1431","Type":"ContainerDied","Data":"ddc38eb51050f69292126e71655c93e83779d53c9750fd708564206016958f2f"} Oct 30 16:40:55.664932 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.664882 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79" event={"ID":"fda3b43f-1c7d-44f5-85ef-e34b9dcd1431","Type":"ContainerDied","Data":"fb9af3d798044a5e064d18bd0f8bd5c4b7de2bcf378392106df078a40aca486d"} Oct 30 16:40:55.677756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.677715 4861 scope.go:122] "RemoveContainer" containerID="871718777fc6b2c493161e23f49e16ae4e8340e88ed07648099d83148c6fd459" Oct 30 16:40:55.678041 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:40:55.678024 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"871718777fc6b2c493161e23f49e16ae4e8340e88ed07648099d83148c6fd459\": container with ID starting with 871718777fc6b2c493161e23f49e16ae4e8340e88ed07648099d83148c6fd459 not found: ID does not exist" containerID="871718777fc6b2c493161e23f49e16ae4e8340e88ed07648099d83148c6fd459" Oct 30 16:40:55.678113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.678047 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871718777fc6b2c493161e23f49e16ae4e8340e88ed07648099d83148c6fd459"} err="failed to get container status \"871718777fc6b2c493161e23f49e16ae4e8340e88ed07648099d83148c6fd459\": rpc error: code = NotFound desc = could not find container \"871718777fc6b2c493161e23f49e16ae4e8340e88ed07648099d83148c6fd459\": container with ID starting with 871718777fc6b2c493161e23f49e16ae4e8340e88ed07648099d83148c6fd459 not found: ID does not exist" Oct 30 16:40:55.678113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.678069 4861 scope.go:122] "RemoveContainer" containerID="ddc38eb51050f69292126e71655c93e83779d53c9750fd708564206016958f2f" Oct 30 16:40:55.687556 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.687540 4861 scope.go:122] "RemoveContainer" containerID="ddc38eb51050f69292126e71655c93e83779d53c9750fd708564206016958f2f" Oct 30 16:40:55.687828 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:40:55.687807 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc38eb51050f69292126e71655c93e83779d53c9750fd708564206016958f2f\": container with ID starting with ddc38eb51050f69292126e71655c93e83779d53c9750fd708564206016958f2f not found: ID does not exist" containerID="ddc38eb51050f69292126e71655c93e83779d53c9750fd708564206016958f2f" Oct 30 16:40:55.687942 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.687837 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc38eb51050f69292126e71655c93e83779d53c9750fd708564206016958f2f"} err="failed to get container status \"ddc38eb51050f69292126e71655c93e83779d53c9750fd708564206016958f2f\": rpc error: code = NotFound desc = could not find container \"ddc38eb51050f69292126e71655c93e83779d53c9750fd708564206016958f2f\": container with ID starting with ddc38eb51050f69292126e71655c93e83779d53c9750fd708564206016958f2f not found: ID does not exist" Oct 30 16:40:55.718295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.718265 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79"] Oct 30 16:40:55.757991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.756136 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8295b2-7b88bfb8c-w9f79"] Oct 30 16:40:55.906748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.901977 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda3b43f-1c7d-44f5-85ef-e34b9dcd1431" path="/var/lib/kubelet/pods/fda3b43f-1c7d-44f5-85ef-e34b9dcd1431/volumes" Oct 30 16:40:55.906748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.902424 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9"] Oct 30 16:40:55.923747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:55.907692 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8295b2-776f959c6b-gghn9"] Oct 30 16:40:56.422916 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:56.422873 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7"] Oct 30 16:40:56.620389 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:56.620351 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:56.771748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:56.771634 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:57.538123 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.538088 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w"] Oct 30 16:40:57.538811 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.538796 4861 cpu_manager.go:414] "containerMap: removing container" podUID="bc8a280e-2bfb-41e2-90ad-f92c9832e30a" containerName="payload-processing" Oct 30 16:40:57.538866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.538812 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc8a280e-2bfb-41e2-90ad-f92c9832e30a" containerName="payload-processing" Oct 30 16:40:57.538866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.538821 4861 cpu_manager.go:414] "containerMap: removing container" podUID="fda3b43f-1c7d-44f5-85ef-e34b9dcd1431" containerName="payload-pre-processing" Oct 30 16:40:57.538866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.538826 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda3b43f-1c7d-44f5-85ef-e34b9dcd1431" containerName="payload-pre-processing" Oct 30 16:40:57.538977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.538944 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc8a280e-2bfb-41e2-90ad-f92c9832e30a" containerName="payload-processing" Oct 30 16:40:57.538977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.538956 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda3b43f-1c7d-44f5-85ef-e34b9dcd1431" containerName="payload-pre-processing" Oct 30 16:40:57.544110 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.544093 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:40:57.551430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.551400 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxxff\" (UniqueName: \"kubernetes.io/projected/051f1a45-22fb-410a-93a8-d43cd5c730f1-kube-api-access-xxxff\") pod \"maas-api-e2e-api-a-7410c7-846774d6dc-v278w\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:40:57.551833 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.551680 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051f1a45-22fb-410a-93a8-d43cd5c730f1-openshift-service-ca\") pod \"maas-api-e2e-api-a-7410c7-846774d6dc-v278w\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:40:57.551833 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.551763 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-metrics-certs\") pod \"maas-api-e2e-api-a-7410c7-846774d6dc-v278w\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:40:57.551960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.551912 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-maas-api-tls\") pod \"maas-api-e2e-api-a-7410c7-846774d6dc-v278w\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:40:57.564473 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.564453 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w"] Oct 30 16:40:57.572434 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.572407 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7410c7-serving-cert\"" Oct 30 16:40:57.646848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.646799 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:57.653415 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.653388 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxxff\" (UniqueName: \"kubernetes.io/projected/051f1a45-22fb-410a-93a8-d43cd5c730f1-kube-api-access-xxxff\") pod \"maas-api-e2e-api-a-7410c7-846774d6dc-v278w\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:40:57.653522 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.653510 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051f1a45-22fb-410a-93a8-d43cd5c730f1-openshift-service-ca\") pod \"maas-api-e2e-api-a-7410c7-846774d6dc-v278w\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:40:57.653562 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.653542 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-metrics-certs\") pod \"maas-api-e2e-api-a-7410c7-846774d6dc-v278w\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:40:57.653604 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.653598 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-maas-api-tls\") pod \"maas-api-e2e-api-a-7410c7-846774d6dc-v278w\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:40:57.653768 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:40:57.653755 4861 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-serving-cert: secret "maas-api-e2e-api-a-7410c7-serving-cert" not found Oct 30 16:40:57.653827 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:40:57.653817 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-maas-api-tls podName:051f1a45-22fb-410a-93a8-d43cd5c730f1 nodeName:}" failed. No retries permitted until 2026-10-30 16:40:58.153801362 +0000 UTC m=+1088.496475377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-maas-api-tls") pod "maas-api-e2e-api-a-7410c7-846774d6dc-v278w" (UID: "051f1a45-22fb-410a-93a8-d43cd5c730f1") : secret "maas-api-e2e-api-a-7410c7-serving-cert" not found Oct 30 16:40:57.654318 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.654296 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051f1a45-22fb-410a-93a8-d43cd5c730f1-openshift-service-ca\") pod \"maas-api-e2e-api-a-7410c7-846774d6dc-v278w\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:40:57.656138 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.656121 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-metrics-certs\") pod \"maas-api-e2e-api-a-7410c7-846774d6dc-v278w\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:40:57.679787 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.679763 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxxff\" (UniqueName: \"kubernetes.io/projected/051f1a45-22fb-410a-93a8-d43cd5c730f1-kube-api-access-xxxff\") pod \"maas-api-e2e-api-a-7410c7-846774d6dc-v278w\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:40:57.796332 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.796143 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:57.869566 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.863837 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc8a280e-2bfb-41e2-90ad-f92c9832e30a" path="/var/lib/kubelet/pods/bc8a280e-2bfb-41e2-90ad-f92c9832e30a/volumes" Oct 30 16:40:57.938087 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:57.938054 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:58.175776 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:58.175668 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-maas-api-tls\") pod \"maas-api-e2e-api-a-7410c7-846774d6dc-v278w\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:40:58.178803 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:58.178778 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-maas-api-tls\") pod \"maas-api-e2e-api-a-7410c7-846774d6dc-v278w\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:40:58.463715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:58.463049 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:40:58.819752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:58.819670 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w"] Oct 30 16:40:59.587559 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:59.587484 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:40:59.705353 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:59.705309 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" event={"ID":"051f1a45-22fb-410a-93a8-d43cd5c730f1","Type":"ContainerStarted","Data":"8b1bd4f5437947ab6140c52c881acacf7c2231e40ac325fb7a78270dfb9123b2"} Oct 30 16:40:59.705559 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:59.705381 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" event={"ID":"051f1a45-22fb-410a-93a8-d43cd5c730f1","Type":"ContainerStarted","Data":"a4aa46d6d0beb7b50572cd842a6ba460bf1ff95da8a3553ccbb5f58d074fb398"} Oct 30 16:40:59.706571 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:59.706547 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:40:59.750596 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:40:59.750551 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" podStartSLOduration=2.339791478 podStartE2EDuration="2.750537356s" totalImagesPullingTime="410.745878ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:59.749082494 +0000 UTC m=+1090.091756532" watchObservedRunningTime="2026-10-30 16:40:59.750537356 +0000 UTC m=+1090.093211421" Oct 30 16:41:01.423616 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:01.423556 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" podUID="5c40832c-edc2-4075-be6f-22547bd79a82" containerName="payload-processing" containerID="cri-o://95ed41af8e4ce15bd5650de6970a3f7d43bb8e5343e1042fe7adece440da3045" gracePeriod=25 Oct 30 16:41:01.441546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:01.441525 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:01.721082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:01.721051 4861 generic.go:356] "Generic (PLEG): container finished" podID="5c40832c-edc2-4075-be6f-22547bd79a82" containerID="95ed41af8e4ce15bd5650de6970a3f7d43bb8e5343e1042fe7adece440da3045" exitCode=0 Oct 30 16:41:01.721218 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:01.721147 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" event={"ID":"5c40832c-edc2-4075-be6f-22547bd79a82","Type":"ContainerDied","Data":"95ed41af8e4ce15bd5650de6970a3f7d43bb8e5343e1042fe7adece440da3045"} Oct 30 16:41:01.721218 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:01.721198 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" event={"ID":"5c40832c-edc2-4075-be6f-22547bd79a82","Type":"ContainerDied","Data":"dab6fbebea75d22950e93d38d99c9a167fa47b22103e5ccd95e3e6333a3cead6"} Oct 30 16:41:01.721310 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:01.721223 4861 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dab6fbebea75d22950e93d38d99c9a167fa47b22103e5ccd95e3e6333a3cead6" Oct 30 16:41:01.731837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:01.731816 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" Oct 30 16:41:01.784112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:01.784082 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c40832c-edc2-4075-be6f-22547bd79a82-kube-api-access-mdlzn\" (UniqueName: \"kubernetes.io/projected/5c40832c-edc2-4075-be6f-22547bd79a82-kube-api-access-mdlzn\") pod \"5c40832c-edc2-4075-be6f-22547bd79a82\" (UID: \"5c40832c-edc2-4075-be6f-22547bd79a82\") " Oct 30 16:41:01.784305 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:01.784206 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c40832c-edc2-4075-be6f-22547bd79a82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c40832c-edc2-4075-be6f-22547bd79a82-plugins-config-volume\") pod \"5c40832c-edc2-4075-be6f-22547bd79a82\" (UID: \"5c40832c-edc2-4075-be6f-22547bd79a82\") " Oct 30 16:41:01.784822 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:01.784797 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c40832c-edc2-4075-be6f-22547bd79a82-plugins-config-volume" pod "5c40832c-edc2-4075-be6f-22547bd79a82" (UID: "5c40832c-edc2-4075-be6f-22547bd79a82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:01.785223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:01.785189 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c40832c-edc2-4075-be6f-22547bd79a82-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:41:01.787273 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:01.787251 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c40832c-edc2-4075-be6f-22547bd79a82-kube-api-access-mdlzn" pod "5c40832c-edc2-4075-be6f-22547bd79a82" (UID: "5c40832c-edc2-4075-be6f-22547bd79a82"). InnerVolumeSpecName "kube-api-access-mdlzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:01.887225 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:01.887205 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdlzn\" (UniqueName: \"kubernetes.io/projected/5c40832c-edc2-4075-be6f-22547bd79a82-kube-api-access-mdlzn\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:41:02.727882 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:02.727854 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7" Oct 30 16:41:02.781894 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:02.781864 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7"] Oct 30 16:41:02.793794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:02.793763 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0b09f1-56b8676d74-fdgv7"] Oct 30 16:41:03.155172 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:03.155040 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:03.808663 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:03.808631 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c40832c-edc2-4075-be6f-22547bd79a82" path="/var/lib/kubelet/pods/5c40832c-edc2-4075-be6f-22547bd79a82/volumes" Oct 30 16:41:04.911318 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:04.911282 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:06.636979 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:06.636946 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:06.746749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:06.738907 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:41:08.380806 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:08.380773 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:10.191122 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:10.191091 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:12.175574 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:12.175539 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:14.001863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:14.001833 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:15.830960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:15.830925 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:17.785540 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:17.785503 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:19.666872 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:19.666844 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:20.290740 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:41:20.290690 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:41:21.280912 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:21.280884 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:21.416214 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:21.416181 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:21.960938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:21.960900 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:22.401177 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:22.401139 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:22.562753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:22.562617 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:23.075080 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:23.075040 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7440-9387f1"] Oct 30 16:41:23.075799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:23.075779 4861 cpu_manager.go:414] "containerMap: removing container" podUID="5c40832c-edc2-4075-be6f-22547bd79a82" containerName="payload-processing" Oct 30 16:41:23.075927 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:23.075802 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c40832c-edc2-4075-be6f-22547bd79a82" containerName="payload-processing" Oct 30 16:41:23.076003 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:23.075987 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c40832c-edc2-4075-be6f-22547bd79a82" containerName="payload-processing" Oct 30 16:41:23.080038 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:23.080017 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7440-9387f1" Oct 30 16:41:23.095631 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:23.095608 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7440-9387f1"] Oct 30 16:41:23.104454 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:23.104436 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qn5rq\"" Oct 30 16:41:23.176493 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:23.176462 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l55fh\" (UniqueName: \"kubernetes.io/projected/84a308d3-b459-4757-8a3b-5e389c10410e-kube-api-access-l55fh\") pod \"test-curl-7440-9387f1\" (UID: \"84a308d3-b459-4757-8a3b-5e389c10410e\") " pod="opendatahub/test-curl-7440-9387f1" Oct 30 16:41:23.278339 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:23.278301 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l55fh\" (UniqueName: \"kubernetes.io/projected/84a308d3-b459-4757-8a3b-5e389c10410e-kube-api-access-l55fh\") pod \"test-curl-7440-9387f1\" (UID: \"84a308d3-b459-4757-8a3b-5e389c10410e\") " pod="opendatahub/test-curl-7440-9387f1" Oct 30 16:41:23.280011 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:23.279983 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:23.366796 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:23.366427 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l55fh\" (UniqueName: \"kubernetes.io/projected/84a308d3-b459-4757-8a3b-5e389c10410e-kube-api-access-l55fh\") pod \"test-curl-7440-9387f1\" (UID: \"84a308d3-b459-4757-8a3b-5e389c10410e\") " pod="opendatahub/test-curl-7440-9387f1" Oct 30 16:41:23.392995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:23.392961 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7440-9387f1" Oct 30 16:41:23.701899 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:23.701869 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7440-9387f1"] Oct 30 16:41:23.702286 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:41:23.702212 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84a308d3_b459_4757_8a3b_5e389c10410e.slice/crio-531126f6469c736992798d29250fa17bdcdfba7464b925d16238a5992fc173f6 WatchSource:0}: Error finding container 531126f6469c736992798d29250fa17bdcdfba7464b925d16238a5992fc173f6: Status 404 returned error can't find the container with id 531126f6469c736992798d29250fa17bdcdfba7464b925d16238a5992fc173f6 Oct 30 16:41:23.919736 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:23.919693 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7440-9387f1" event={"ID":"84a308d3-b459-4757-8a3b-5e389c10410e","Type":"ContainerStarted","Data":"531126f6469c736992798d29250fa17bdcdfba7464b925d16238a5992fc173f6"} Oct 30 16:41:25.240822 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:25.240780 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:25.936853 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:25.936711 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7440-9387f1" event={"ID":"84a308d3-b459-4757-8a3b-5e389c10410e","Type":"ContainerStarted","Data":"b201c83f834cd59b83474cdadd2208b5615e5a51fc9348c1e3d8e5d4ac9c4b71"} Oct 30 16:41:25.978804 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:25.978715 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7440-9387f1" podStartSLOduration=1.031828156 podStartE2EDuration="2.9787015s" totalImagesPullingTime="1.946873344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:25.975286766 +0000 UTC m=+1116.317960803" watchObservedRunningTime="2026-10-30 16:41:25.9787015 +0000 UTC m=+1116.321375538" Oct 30 16:41:26.322695 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:26.322665 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:26.461406 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:26.461368 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:26.852790 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:26.852755 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7440-9387f1"] Oct 30 16:41:26.862501 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:26.862476 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7440-9387f1"] Oct 30 16:41:27.246234 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:27.246198 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:27.950984 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:27.950951 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7440-9387f1" podUID="84a308d3-b459-4757-8a3b-5e389c10410e" containerName="test-curl-7440-9387f1" containerID="cri-o://b201c83f834cd59b83474cdadd2208b5615e5a51fc9348c1e3d8e5d4ac9c4b71" gracePeriod=2 Oct 30 16:41:28.103260 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.103228 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:28.247547 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.247509 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:28.256115 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.256089 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w"] Oct 30 16:41:28.256879 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.256857 4861 cpu_manager.go:414] "containerMap: removing container" podUID="84a308d3-b459-4757-8a3b-5e389c10410e" containerName="test-curl-7440-9387f1" Oct 30 16:41:28.256879 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.256879 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a308d3-b459-4757-8a3b-5e389c10410e" containerName="test-curl-7440-9387f1" Oct 30 16:41:28.257112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.257071 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a308d3-b459-4757-8a3b-5e389c10410e" containerName="test-curl-7440-9387f1" Oct 30 16:41:28.261208 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.261190 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" Oct 30 16:41:28.281074 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.281053 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w"] Oct 30 16:41:28.288905 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.288873 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1189d4\"" Oct 30 16:41:28.289207 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.289177 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1189d4-dockercfg-n7pzv\"" Oct 30 16:41:28.431637 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.404697 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txx5m\" (UniqueName: \"kubernetes.io/projected/a8f9abdc-9f33-4065-abac-a3fbf8ec8504-kube-api-access-txx5m\") pod \"payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w\" (UID: \"a8f9abdc-9f33-4065-abac-a3fbf8ec8504\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" Oct 30 16:41:28.431637 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.404814 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f9abdc-9f33-4065-abac-a3fbf8ec8504-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w\" (UID: \"a8f9abdc-9f33-4065-abac-a3fbf8ec8504\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" Oct 30 16:41:28.507918 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.507820 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txx5m\" (UniqueName: \"kubernetes.io/projected/a8f9abdc-9f33-4065-abac-a3fbf8ec8504-kube-api-access-txx5m\") pod \"payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w\" (UID: \"a8f9abdc-9f33-4065-abac-a3fbf8ec8504\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" Oct 30 16:41:28.508295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.508267 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f9abdc-9f33-4065-abac-a3fbf8ec8504-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w\" (UID: \"a8f9abdc-9f33-4065-abac-a3fbf8ec8504\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" Oct 30 16:41:28.510672 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.510637 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f9abdc-9f33-4065-abac-a3fbf8ec8504-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w\" (UID: \"a8f9abdc-9f33-4065-abac-a3fbf8ec8504\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" Oct 30 16:41:28.560747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.556445 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txx5m\" (UniqueName: \"kubernetes.io/projected/a8f9abdc-9f33-4065-abac-a3fbf8ec8504-kube-api-access-txx5m\") pod \"payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w\" (UID: \"a8f9abdc-9f33-4065-abac-a3fbf8ec8504\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" Oct 30 16:41:28.575228 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.574964 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" Oct 30 16:41:28.963299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:28.963271 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w"] Oct 30 16:41:29.541319 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:29.533894 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:29.967460 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:29.967430 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" event={"ID":"a8f9abdc-9f33-4065-abac-a3fbf8ec8504","Type":"ContainerStarted","Data":"b9eadfd8f7e7967deeb599d9d8012f273f1585cb6cff12a7785a51bd622ed677"} Oct 30 16:41:29.967866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:29.967481 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" event={"ID":"a8f9abdc-9f33-4065-abac-a3fbf8ec8504","Type":"ContainerStarted","Data":"838381f1fc27a19e55d76fb7b6f1c8d3e7fcb7970d8d64db8eec1145519562d9"} Oct 30 16:41:29.967866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:29.967676 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" Oct 30 16:41:30.021190 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:30.019659 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" podStartSLOduration=2.019638433 podStartE2EDuration="2.019638433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:30.019352416 +0000 UTC m=+1120.362026454" watchObservedRunningTime="2026-10-30 16:41:30.019638433 +0000 UTC m=+1120.362312474" Oct 30 16:41:30.350984 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:30.350961 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7440-9387f1" Oct 30 16:41:30.373401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:30.373343 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a308d3-b459-4757-8a3b-5e389c10410e-kube-api-access-l55fh\" (UniqueName: \"kubernetes.io/projected/84a308d3-b459-4757-8a3b-5e389c10410e-kube-api-access-l55fh\") pod \"84a308d3-b459-4757-8a3b-5e389c10410e\" (UID: \"84a308d3-b459-4757-8a3b-5e389c10410e\") " Oct 30 16:41:30.375608 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:30.375587 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a308d3-b459-4757-8a3b-5e389c10410e-kube-api-access-l55fh" pod "84a308d3-b459-4757-8a3b-5e389c10410e" (UID: "84a308d3-b459-4757-8a3b-5e389c10410e"). InnerVolumeSpecName "kube-api-access-l55fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:30.476001 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:30.475964 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l55fh\" (UniqueName: \"kubernetes.io/projected/84a308d3-b459-4757-8a3b-5e389c10410e-kube-api-access-l55fh\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:41:30.976905 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:30.976874 4861 generic.go:356] "Generic (PLEG): container finished" podID="84a308d3-b459-4757-8a3b-5e389c10410e" containerID="b201c83f834cd59b83474cdadd2208b5615e5a51fc9348c1e3d8e5d4ac9c4b71" exitCode=137 Oct 30 16:41:30.977320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:30.976915 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7440-9387f1" Oct 30 16:41:30.977320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:30.976966 4861 scope.go:122] "RemoveContainer" containerID="b201c83f834cd59b83474cdadd2208b5615e5a51fc9348c1e3d8e5d4ac9c4b71" Oct 30 16:41:30.988224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:30.988195 4861 scope.go:122] "RemoveContainer" containerID="b201c83f834cd59b83474cdadd2208b5615e5a51fc9348c1e3d8e5d4ac9c4b71" Oct 30 16:41:30.988515 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:41:30.988499 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b201c83f834cd59b83474cdadd2208b5615e5a51fc9348c1e3d8e5d4ac9c4b71\": container with ID starting with b201c83f834cd59b83474cdadd2208b5615e5a51fc9348c1e3d8e5d4ac9c4b71 not found: ID does not exist" containerID="b201c83f834cd59b83474cdadd2208b5615e5a51fc9348c1e3d8e5d4ac9c4b71" Oct 30 16:41:30.988599 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:30.988526 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b201c83f834cd59b83474cdadd2208b5615e5a51fc9348c1e3d8e5d4ac9c4b71"} err="failed to get container status \"b201c83f834cd59b83474cdadd2208b5615e5a51fc9348c1e3d8e5d4ac9c4b71\": rpc error: code = NotFound desc = could not find container \"b201c83f834cd59b83474cdadd2208b5615e5a51fc9348c1e3d8e5d4ac9c4b71\": container with ID starting with b201c83f834cd59b83474cdadd2208b5615e5a51fc9348c1e3d8e5d4ac9c4b71 not found: ID does not exist" Oct 30 16:41:30.999834 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:41:30.999809 4861 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7440-9387f1\" is forbidden: User \"system:node:ip-10-0-131-179.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-179.ec2.internal' and this object" podUID="84a308d3-b459-4757-8a3b-5e389c10410e" pod="opendatahub/test-curl-7440-9387f1" Oct 30 16:41:31.410253 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:31.410167 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:31.808159 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:31.808121 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a308d3-b459-4757-8a3b-5e389c10410e" path="/var/lib/kubelet/pods/84a308d3-b459-4757-8a3b-5e389c10410e/volumes" Oct 30 16:41:33.045045 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:33.045013 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:33.985058 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:33.985031 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:34.215139 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:34.215108 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:34.365579 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:34.365482 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:34.661889 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:34.661812 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:35.965788 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:35.965753 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5"] Oct 30 16:41:35.966287 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:35.966021 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" podUID="5bdd45e6-87c4-4168-9bb3-9a627cbf5813" containerName="maas-api" containerID="cri-o://5dbe3cc5b61bd43ffa717c68fbf502747969dafe11e9b153b5b955abdb5901ee" gracePeriod=30 Oct 30 16:41:36.054740 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:36.054690 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:36.421141 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:36.421108 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:41:36.551369 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:36.551325 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-maas-api-tls\") pod \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " Oct 30 16:41:36.551571 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:36.551394 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-kube-api-access-cpvmh\" (UniqueName: \"kubernetes.io/projected/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-kube-api-access-cpvmh\") pod \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " Oct 30 16:41:36.551778 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:36.551706 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-metrics-certs\") pod \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " Oct 30 16:41:36.551879 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:36.551841 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-openshift-service-ca\") pod \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\" (UID: \"5bdd45e6-87c4-4168-9bb3-9a627cbf5813\") " Oct 30 16:41:36.561797 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:36.553260 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-openshift-service-ca" pod "5bdd45e6-87c4-4168-9bb3-9a627cbf5813" (UID: "5bdd45e6-87c4-4168-9bb3-9a627cbf5813"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:36.610703 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:36.610654 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-metrics-certs" pod "5bdd45e6-87c4-4168-9bb3-9a627cbf5813" (UID: "5bdd45e6-87c4-4168-9bb3-9a627cbf5813"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:41:36.631756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:36.625242 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-kube-api-access-cpvmh" pod "5bdd45e6-87c4-4168-9bb3-9a627cbf5813" (UID: "5bdd45e6-87c4-4168-9bb3-9a627cbf5813"). InnerVolumeSpecName "kube-api-access-cpvmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:36.640614 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:36.632041 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-maas-api-tls" pod "5bdd45e6-87c4-4168-9bb3-9a627cbf5813" (UID: "5bdd45e6-87c4-4168-9bb3-9a627cbf5813"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:41:36.708441 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:36.668231 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:41:36.708441 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:36.668266 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:41:36.708441 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:36.668282 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpvmh\" (UniqueName: \"kubernetes.io/projected/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-kube-api-access-cpvmh\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:41:36.708441 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:36.668295 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5bdd45e6-87c4-4168-9bb3-9a627cbf5813-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:41:36.923635 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:36.923546 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:37.051399 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:37.051365 4861 generic.go:356] "Generic (PLEG): container finished" podID="5bdd45e6-87c4-4168-9bb3-9a627cbf5813" containerID="5dbe3cc5b61bd43ffa717c68fbf502747969dafe11e9b153b5b955abdb5901ee" exitCode=0 Oct 30 16:41:37.051865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:37.051413 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" Oct 30 16:41:37.051865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:37.051452 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" event={"ID":"5bdd45e6-87c4-4168-9bb3-9a627cbf5813","Type":"ContainerDied","Data":"5dbe3cc5b61bd43ffa717c68fbf502747969dafe11e9b153b5b955abdb5901ee"} Oct 30 16:41:37.051865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:37.051511 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" event={"ID":"5bdd45e6-87c4-4168-9bb3-9a627cbf5813","Type":"ContainerDied","Data":"7f67d853f443569e59427900376f0345e932765fdd7eae3a70c9f4fec79529ce"} Oct 30 16:41:37.051865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:37.051542 4861 scope.go:122] "RemoveContainer" containerID="5dbe3cc5b61bd43ffa717c68fbf502747969dafe11e9b153b5b955abdb5901ee" Oct 30 16:41:37.065884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:37.065691 4861 scope.go:122] "RemoveContainer" containerID="5dbe3cc5b61bd43ffa717c68fbf502747969dafe11e9b153b5b955abdb5901ee" Oct 30 16:41:37.066090 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:41:37.066051 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dbe3cc5b61bd43ffa717c68fbf502747969dafe11e9b153b5b955abdb5901ee\": container with ID starting with 5dbe3cc5b61bd43ffa717c68fbf502747969dafe11e9b153b5b955abdb5901ee not found: ID does not exist" containerID="5dbe3cc5b61bd43ffa717c68fbf502747969dafe11e9b153b5b955abdb5901ee" Oct 30 16:41:37.066090 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:37.066075 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dbe3cc5b61bd43ffa717c68fbf502747969dafe11e9b153b5b955abdb5901ee"} err="failed to get container status \"5dbe3cc5b61bd43ffa717c68fbf502747969dafe11e9b153b5b955abdb5901ee\": rpc error: code = NotFound desc = could not find container \"5dbe3cc5b61bd43ffa717c68fbf502747969dafe11e9b153b5b955abdb5901ee\": container with ID starting with 5dbe3cc5b61bd43ffa717c68fbf502747969dafe11e9b153b5b955abdb5901ee not found: ID does not exist" Oct 30 16:41:37.067964 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:37.067943 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:37.106220 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:37.106195 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5"] Oct 30 16:41:37.172751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:37.170056 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5"] Oct 30 16:41:37.233759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:37.233450 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:37.549935 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:37.549860 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:37.809582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:37.809505 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bdd45e6-87c4-4168-9bb3-9a627cbf5813" path="/var/lib/kubelet/pods/5bdd45e6-87c4-4168-9bb3-9a627cbf5813/volumes" Oct 30 16:41:38.447503 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:38.447468 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq"] Oct 30 16:41:38.478785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:38.478756 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s"] Oct 30 16:41:38.620879 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:38.620851 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:38.772747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:38.767446 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:39.163161 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:39.163082 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:39.305748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:39.302829 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0b09f1-f44697ff-pqgb5" podUID="5bdd45e6-87c4-4168-9bb3-9a627cbf5813" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.93:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:40.980876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:40.980848 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" Oct 30 16:41:41.205098 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:41.205068 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:41.355748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:41.355610 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:41.675459 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:41.675334 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:42.761100 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:42.761066 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:43.189968 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:43.189844 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:43.448824 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:43.448738 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" podUID="50d1af4a-8f20-4200-85b7-615c9475d573" containerName="payload-processing" containerID="cri-o://c375ccc76ffe115d26f321dacd7500c0c4f8eccb5f621f6fda43fd6837287524" gracePeriod=25 Oct 30 16:41:43.479286 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:43.479251 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" podUID="57333841-13b5-466b-8fc5-bf4c0b5ea649" containerName="payload-pre-processing" containerID="cri-o://3552dbb28daa4e46d3c8dcde4eb8c572563a253df3a8308721968bc21f53460e" gracePeriod=25 Oct 30 16:41:44.387462 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:44.387382 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:44.532539 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:44.532506 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:44.716260 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:44.716179 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:45.062187 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.062142 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" podUID="57333841-13b5-466b-8fc5-bf4c0b5ea649" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:41:45.062341 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.062211 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" podUID="50d1af4a-8f20-4200-85b7-615c9475d573" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:41:45.157225 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.157181 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm"] Oct 30 16:41:45.158328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.158300 4861 cpu_manager.go:414] "containerMap: removing container" podUID="5bdd45e6-87c4-4168-9bb3-9a627cbf5813" containerName="maas-api" Oct 30 16:41:45.158328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.158327 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bdd45e6-87c4-4168-9bb3-9a627cbf5813" containerName="maas-api" Oct 30 16:41:45.158541 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.158513 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bdd45e6-87c4-4168-9bb3-9a627cbf5813" containerName="maas-api" Oct 30 16:41:45.163825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.163805 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:45.175532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.174183 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-metrics-certs\") pod \"maas-api-e2e-api-b-1189d4-868d76874f-j2jkm\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:45.175532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.174292 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c265c02-98e3-4f69-b447-0811f146b548-openshift-service-ca\") pod \"maas-api-e2e-api-b-1189d4-868d76874f-j2jkm\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:45.175532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.174585 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-maas-api-tls\") pod \"maas-api-e2e-api-b-1189d4-868d76874f-j2jkm\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:45.175532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.174624 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w79cc\" (UniqueName: \"kubernetes.io/projected/5c265c02-98e3-4f69-b447-0811f146b548-kube-api-access-w79cc\") pod \"maas-api-e2e-api-b-1189d4-868d76874f-j2jkm\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:45.182998 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.182958 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm"] Oct 30 16:41:45.191998 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.191975 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1189d4-serving-cert\"" Oct 30 16:41:45.275968 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.275938 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-maas-api-tls\") pod \"maas-api-e2e-api-b-1189d4-868d76874f-j2jkm\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:45.276164 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.275976 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w79cc\" (UniqueName: \"kubernetes.io/projected/5c265c02-98e3-4f69-b447-0811f146b548-kube-api-access-w79cc\") pod \"maas-api-e2e-api-b-1189d4-868d76874f-j2jkm\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:45.276164 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.276039 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-metrics-certs\") pod \"maas-api-e2e-api-b-1189d4-868d76874f-j2jkm\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:45.276164 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.276147 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c265c02-98e3-4f69-b447-0811f146b548-openshift-service-ca\") pod \"maas-api-e2e-api-b-1189d4-868d76874f-j2jkm\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:45.276386 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:41:45.276328 4861 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-serving-cert: secret "maas-api-e2e-api-b-1189d4-serving-cert" not found Oct 30 16:41:45.276445 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:41:45.276418 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-maas-api-tls podName:5c265c02-98e3-4f69-b447-0811f146b548 nodeName:}" failed. No retries permitted until 2026-10-30 16:41:45.776395255 +0000 UTC m=+1136.119069285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-maas-api-tls") pod "maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" (UID: "5c265c02-98e3-4f69-b447-0811f146b548") : secret "maas-api-e2e-api-b-1189d4-serving-cert" not found Oct 30 16:41:45.276896 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.276871 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c265c02-98e3-4f69-b447-0811f146b548-openshift-service-ca\") pod \"maas-api-e2e-api-b-1189d4-868d76874f-j2jkm\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:45.280254 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.280229 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-metrics-certs\") pod \"maas-api-e2e-api-b-1189d4-868d76874f-j2jkm\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:45.284023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.283986 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:45.302672 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.302646 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w79cc\" (UniqueName: \"kubernetes.io/projected/5c265c02-98e3-4f69-b447-0811f146b548-kube-api-access-w79cc\") pod \"maas-api-e2e-api-b-1189d4-868d76874f-j2jkm\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:45.437800 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.435995 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:45.794539 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.794504 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-maas-api-tls\") pod \"maas-api-e2e-api-b-1189d4-868d76874f-j2jkm\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:45.797080 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:45.797054 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-maas-api-tls\") pod \"maas-api-e2e-api-b-1189d4-868d76874f-j2jkm\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:46.081605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:46.081528 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:46.398623 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:46.398597 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm"] Oct 30 16:41:46.400390 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:41:46.400361 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c265c02_98e3_4f69_b447_0811f146b548.slice/crio-3ceff93499f6b350f17346417469e7a01d74651efef40d5998905ee07884800b WatchSource:0}: Error finding container 3ceff93499f6b350f17346417469e7a01d74651efef40d5998905ee07884800b: Status 404 returned error can't find the container with id 3ceff93499f6b350f17346417469e7a01d74651efef40d5998905ee07884800b Oct 30 16:41:47.128866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:47.128761 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" event={"ID":"5c265c02-98e3-4f69-b447-0811f146b548","Type":"ContainerStarted","Data":"fb343a08804294fc5013b6814c4c0747c92c03aa0549433a2cf122311826c7ba"} Oct 30 16:41:47.128866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:47.128828 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" event={"ID":"5c265c02-98e3-4f69-b447-0811f146b548","Type":"ContainerStarted","Data":"3ceff93499f6b350f17346417469e7a01d74651efef40d5998905ee07884800b"} Oct 30 16:41:47.128866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:47.128863 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:47.190066 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:47.190015 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" podStartSLOduration=1.798327031 podStartE2EDuration="2.189999334s" totalImagesPullingTime="391.672303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:47.188315298 +0000 UTC m=+1137.530989336" watchObservedRunningTime="2026-10-30 16:41:47.189999334 +0000 UTC m=+1137.532673371" Oct 30 16:41:47.432899 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:47.432826 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:49.357506 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:49.357476 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:50.289930 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:41:50.289896 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:41:51.222346 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:51.222310 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:52.559906 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:52.559872 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng"] Oct 30 16:41:52.592567 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:52.592535 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs"] Oct 30 16:41:52.734768 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:52.734733 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:52.886268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:52.886180 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:53.033388 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:53.033354 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:53.141684 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:53.141609 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:41:53.745114 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:53.745080 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:53.897469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:53.897436 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:54.877566 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:54.877531 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:55.062386 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:55.062342 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" podUID="50d1af4a-8f20-4200-85b7-615c9475d573" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:41:55.062564 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:55.062338 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" podUID="57333841-13b5-466b-8fc5-bf4c0b5ea649" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:41:55.706856 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:55.706825 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d"] Oct 30 16:41:55.740442 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:55.740416 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p"] Oct 30 16:41:55.879368 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:55.879342 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:56.026934 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:56.026889 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:56.876254 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:56.876226 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:57.022263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:57.022222 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:57.561031 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:57.560996 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" podUID="c44a2b96-5f28-4bee-89b1-3aa1967fdad8" containerName="payload-processing" containerID="cri-o://6befaf433708f8636586cac4edd7463b19c64346a7e8afb37274d0ef7cf9a230" gracePeriod=25 Oct 30 16:41:57.593703 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:57.593670 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" podUID="34c86fa8-9c6e-4aac-92e4-434250b1b3a0" containerName="payload-pre-processing" containerID="cri-o://fe5fea3d3a28d7b553df7b4213144c93cd0719dc7cbf1c9120d88de26bd16407" gracePeriod=25 Oct 30 16:41:57.874560 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:57.874528 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" Oct 30 16:41:57.917267 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:57.917240 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" Oct 30 16:41:57.936203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:57.936175 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c44a2b96-5f28-4bee-89b1-3aa1967fdad8-kube-api-access-c5h8j\" (UniqueName: \"kubernetes.io/projected/c44a2b96-5f28-4bee-89b1-3aa1967fdad8-kube-api-access-c5h8j\") pod \"c44a2b96-5f28-4bee-89b1-3aa1967fdad8\" (UID: \"c44a2b96-5f28-4bee-89b1-3aa1967fdad8\") " Oct 30 16:41:57.936370 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:57.936300 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c86fa8-9c6e-4aac-92e4-434250b1b3a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c86fa8-9c6e-4aac-92e4-434250b1b3a0-plugins-config-volume\") pod \"34c86fa8-9c6e-4aac-92e4-434250b1b3a0\" (UID: \"34c86fa8-9c6e-4aac-92e4-434250b1b3a0\") " Oct 30 16:41:57.936370 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:57.936345 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c86fa8-9c6e-4aac-92e4-434250b1b3a0-kube-api-access-jblvk\" (UniqueName: \"kubernetes.io/projected/34c86fa8-9c6e-4aac-92e4-434250b1b3a0-kube-api-access-jblvk\") pod \"34c86fa8-9c6e-4aac-92e4-434250b1b3a0\" (UID: \"34c86fa8-9c6e-4aac-92e4-434250b1b3a0\") " Oct 30 16:41:57.936490 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:57.936377 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c44a2b96-5f28-4bee-89b1-3aa1967fdad8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44a2b96-5f28-4bee-89b1-3aa1967fdad8-plugins-config-volume\") pod \"c44a2b96-5f28-4bee-89b1-3aa1967fdad8\" (UID: \"c44a2b96-5f28-4bee-89b1-3aa1967fdad8\") " Oct 30 16:41:57.936968 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:57.936942 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c86fa8-9c6e-4aac-92e4-434250b1b3a0-plugins-config-volume" pod "34c86fa8-9c6e-4aac-92e4-434250b1b3a0" (UID: "34c86fa8-9c6e-4aac-92e4-434250b1b3a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:57.937638 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:57.937593 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44a2b96-5f28-4bee-89b1-3aa1967fdad8-plugins-config-volume" pod "c44a2b96-5f28-4bee-89b1-3aa1967fdad8" (UID: "c44a2b96-5f28-4bee-89b1-3aa1967fdad8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:41:57.938861 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:57.938831 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c86fa8-9c6e-4aac-92e4-434250b1b3a0-kube-api-access-jblvk" pod "34c86fa8-9c6e-4aac-92e4-434250b1b3a0" (UID: "34c86fa8-9c6e-4aac-92e4-434250b1b3a0"). InnerVolumeSpecName "kube-api-access-jblvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:57.939065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:57.939044 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44a2b96-5f28-4bee-89b1-3aa1967fdad8-kube-api-access-c5h8j" pod "c44a2b96-5f28-4bee-89b1-3aa1967fdad8" (UID: "c44a2b96-5f28-4bee-89b1-3aa1967fdad8"). InnerVolumeSpecName "kube-api-access-c5h8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:41:58.038692 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.038660 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5h8j\" (UniqueName: \"kubernetes.io/projected/c44a2b96-5f28-4bee-89b1-3aa1967fdad8-kube-api-access-c5h8j\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:41:58.038692 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.038689 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34c86fa8-9c6e-4aac-92e4-434250b1b3a0-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:41:58.038692 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.038698 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jblvk\" (UniqueName: \"kubernetes.io/projected/34c86fa8-9c6e-4aac-92e4-434250b1b3a0-kube-api-access-jblvk\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:41:58.039146 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.038707 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c44a2b96-5f28-4bee-89b1-3aa1967fdad8-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:41:58.208309 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.208217 4861 generic.go:356] "Generic (PLEG): container finished" podID="34c86fa8-9c6e-4aac-92e4-434250b1b3a0" containerID="fe5fea3d3a28d7b553df7b4213144c93cd0719dc7cbf1c9120d88de26bd16407" exitCode=0 Oct 30 16:41:58.208309 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.208271 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" Oct 30 16:41:58.208528 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.208301 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" event={"ID":"34c86fa8-9c6e-4aac-92e4-434250b1b3a0","Type":"ContainerDied","Data":"fe5fea3d3a28d7b553df7b4213144c93cd0719dc7cbf1c9120d88de26bd16407"} Oct 30 16:41:58.208528 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.208360 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs" event={"ID":"34c86fa8-9c6e-4aac-92e4-434250b1b3a0","Type":"ContainerDied","Data":"ce6b73303f890a163624ea462989232819f72491744f7f8ccc5b324fcf5db2db"} Oct 30 16:41:58.208528 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.208389 4861 scope.go:122] "RemoveContainer" containerID="fe5fea3d3a28d7b553df7b4213144c93cd0719dc7cbf1c9120d88de26bd16407" Oct 30 16:41:58.209951 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.209929 4861 generic.go:356] "Generic (PLEG): container finished" podID="c44a2b96-5f28-4bee-89b1-3aa1967fdad8" containerID="6befaf433708f8636586cac4edd7463b19c64346a7e8afb37274d0ef7cf9a230" exitCode=0 Oct 30 16:41:58.210061 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.209980 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" Oct 30 16:41:58.210120 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.210081 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" event={"ID":"c44a2b96-5f28-4bee-89b1-3aa1967fdad8","Type":"ContainerDied","Data":"6befaf433708f8636586cac4edd7463b19c64346a7e8afb37274d0ef7cf9a230"} Oct 30 16:41:58.210166 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.210131 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng" event={"ID":"c44a2b96-5f28-4bee-89b1-3aa1967fdad8","Type":"ContainerDied","Data":"ffc167143f14886a2c94fdfb8bc1b920b06ee74dbcadf070b626cbd1197ff910"} Oct 30 16:41:58.220600 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.220461 4861 scope.go:122] "RemoveContainer" containerID="fe5fea3d3a28d7b553df7b4213144c93cd0719dc7cbf1c9120d88de26bd16407" Oct 30 16:41:58.220971 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:41:58.220952 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5fea3d3a28d7b553df7b4213144c93cd0719dc7cbf1c9120d88de26bd16407\": container with ID starting with fe5fea3d3a28d7b553df7b4213144c93cd0719dc7cbf1c9120d88de26bd16407 not found: ID does not exist" containerID="fe5fea3d3a28d7b553df7b4213144c93cd0719dc7cbf1c9120d88de26bd16407" Oct 30 16:41:58.221050 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.220977 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5fea3d3a28d7b553df7b4213144c93cd0719dc7cbf1c9120d88de26bd16407"} err="failed to get container status \"fe5fea3d3a28d7b553df7b4213144c93cd0719dc7cbf1c9120d88de26bd16407\": rpc error: code = NotFound desc = could not find container \"fe5fea3d3a28d7b553df7b4213144c93cd0719dc7cbf1c9120d88de26bd16407\": container with ID starting with fe5fea3d3a28d7b553df7b4213144c93cd0719dc7cbf1c9120d88de26bd16407 not found: ID does not exist" Oct 30 16:41:58.221050 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.220993 4861 scope.go:122] "RemoveContainer" containerID="6befaf433708f8636586cac4edd7463b19c64346a7e8afb37274d0ef7cf9a230" Oct 30 16:41:58.230326 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.230308 4861 scope.go:122] "RemoveContainer" containerID="6befaf433708f8636586cac4edd7463b19c64346a7e8afb37274d0ef7cf9a230" Oct 30 16:41:58.230598 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:41:58.230575 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6befaf433708f8636586cac4edd7463b19c64346a7e8afb37274d0ef7cf9a230\": container with ID starting with 6befaf433708f8636586cac4edd7463b19c64346a7e8afb37274d0ef7cf9a230 not found: ID does not exist" containerID="6befaf433708f8636586cac4edd7463b19c64346a7e8afb37274d0ef7cf9a230" Oct 30 16:41:58.230687 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.230603 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6befaf433708f8636586cac4edd7463b19c64346a7e8afb37274d0ef7cf9a230"} err="failed to get container status \"6befaf433708f8636586cac4edd7463b19c64346a7e8afb37274d0ef7cf9a230\": rpc error: code = NotFound desc = could not find container \"6befaf433708f8636586cac4edd7463b19c64346a7e8afb37274d0ef7cf9a230\": container with ID starting with 6befaf433708f8636586cac4edd7463b19c64346a7e8afb37274d0ef7cf9a230 not found: ID does not exist" Oct 30 16:41:58.262782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.262760 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs"] Oct 30 16:41:58.267823 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.267801 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-90df54-bf9d9874c-7tggs"] Oct 30 16:41:58.301324 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.301297 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng"] Oct 30 16:41:58.305279 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.305258 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-90df54-5dbf5858b4-ml7ng"] Oct 30 16:41:58.491508 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:58.491480 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:59.173048 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:59.172982 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:59.360210 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:59.360180 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:59.545740 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:59.545703 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:41:59.808215 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:59.808144 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c86fa8-9c6e-4aac-92e4-434250b1b3a0" path="/var/lib/kubelet/pods/34c86fa8-9c6e-4aac-92e4-434250b1b3a0/volumes" Oct 30 16:41:59.808603 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:41:59.808590 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c44a2b96-5f28-4bee-89b1-3aa1967fdad8" path="/var/lib/kubelet/pods/c44a2b96-5f28-4bee-89b1-3aa1967fdad8/volumes" Oct 30 16:42:00.707736 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:00.707684 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" podUID="29d599d5-d4b9-4708-ba37-beecd9e6cfbe" containerName="payload-processing" containerID="cri-o://3a2622d9487c9b642d43051ef2cc61dc20ecc4c06f59ddf87738b709444f9880" gracePeriod=25 Oct 30 16:42:00.740780 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:00.740751 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" podUID="143c2049-fabb-4d96-8c30-6c11b24b2c4c" containerName="payload-pre-processing" containerID="cri-o://9a4f1dd68dd0f457625add2d202c132570b82072f34cf6d3671b0992b75493ac" gracePeriod=25 Oct 30 16:42:01.055512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.055478 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" Oct 30 16:42:01.088794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.088770 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" Oct 30 16:42:01.128024 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.127997 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d599d5-d4b9-4708-ba37-beecd9e6cfbe-kube-api-access-2pntm\" (UniqueName: \"kubernetes.io/projected/29d599d5-d4b9-4708-ba37-beecd9e6cfbe-kube-api-access-2pntm\") pod \"29d599d5-d4b9-4708-ba37-beecd9e6cfbe\" (UID: \"29d599d5-d4b9-4708-ba37-beecd9e6cfbe\") " Oct 30 16:42:01.128235 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.128070 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d599d5-d4b9-4708-ba37-beecd9e6cfbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d599d5-d4b9-4708-ba37-beecd9e6cfbe-plugins-config-volume\") pod \"29d599d5-d4b9-4708-ba37-beecd9e6cfbe\" (UID: \"29d599d5-d4b9-4708-ba37-beecd9e6cfbe\") " Oct 30 16:42:01.128307 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.128255 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/143c2049-fabb-4d96-8c30-6c11b24b2c4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143c2049-fabb-4d96-8c30-6c11b24b2c4c-plugins-config-volume\") pod \"143c2049-fabb-4d96-8c30-6c11b24b2c4c\" (UID: \"143c2049-fabb-4d96-8c30-6c11b24b2c4c\") " Oct 30 16:42:01.128367 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.128321 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143c2049-fabb-4d96-8c30-6c11b24b2c4c-kube-api-access-mzcsh\" (UniqueName: \"kubernetes.io/projected/143c2049-fabb-4d96-8c30-6c11b24b2c4c-kube-api-access-mzcsh\") pod \"143c2049-fabb-4d96-8c30-6c11b24b2c4c\" (UID: \"143c2049-fabb-4d96-8c30-6c11b24b2c4c\") " Oct 30 16:42:01.128714 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.128685 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d599d5-d4b9-4708-ba37-beecd9e6cfbe-plugins-config-volume" pod "29d599d5-d4b9-4708-ba37-beecd9e6cfbe" (UID: "29d599d5-d4b9-4708-ba37-beecd9e6cfbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:01.128846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.128711 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143c2049-fabb-4d96-8c30-6c11b24b2c4c-plugins-config-volume" pod "143c2049-fabb-4d96-8c30-6c11b24b2c4c" (UID: "143c2049-fabb-4d96-8c30-6c11b24b2c4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:01.128946 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.128926 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29d599d5-d4b9-4708-ba37-beecd9e6cfbe-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:01.128998 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.128953 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143c2049-fabb-4d96-8c30-6c11b24b2c4c-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:01.130926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.130886 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143c2049-fabb-4d96-8c30-6c11b24b2c4c-kube-api-access-mzcsh" pod "143c2049-fabb-4d96-8c30-6c11b24b2c4c" (UID: "143c2049-fabb-4d96-8c30-6c11b24b2c4c"). InnerVolumeSpecName "kube-api-access-mzcsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:01.131263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.131246 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d599d5-d4b9-4708-ba37-beecd9e6cfbe-kube-api-access-2pntm" pod "29d599d5-d4b9-4708-ba37-beecd9e6cfbe" (UID: "29d599d5-d4b9-4708-ba37-beecd9e6cfbe"). InnerVolumeSpecName "kube-api-access-2pntm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:01.230351 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.230266 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzcsh\" (UniqueName: \"kubernetes.io/projected/143c2049-fabb-4d96-8c30-6c11b24b2c4c-kube-api-access-mzcsh\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:01.230351 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.230302 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pntm\" (UniqueName: \"kubernetes.io/projected/29d599d5-d4b9-4708-ba37-beecd9e6cfbe-kube-api-access-2pntm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:01.234207 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.234174 4861 generic.go:356] "Generic (PLEG): container finished" podID="29d599d5-d4b9-4708-ba37-beecd9e6cfbe" containerID="3a2622d9487c9b642d43051ef2cc61dc20ecc4c06f59ddf87738b709444f9880" exitCode=0 Oct 30 16:42:01.234411 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.234220 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" Oct 30 16:42:01.234411 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.234256 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" event={"ID":"29d599d5-d4b9-4708-ba37-beecd9e6cfbe","Type":"ContainerDied","Data":"3a2622d9487c9b642d43051ef2cc61dc20ecc4c06f59ddf87738b709444f9880"} Oct 30 16:42:01.234411 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.234306 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d" event={"ID":"29d599d5-d4b9-4708-ba37-beecd9e6cfbe","Type":"ContainerDied","Data":"c6415d329e771495f4538b65d28875f1b605f93240a6a30694bf4bb960fe84a9"} Oct 30 16:42:01.234411 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.234334 4861 scope.go:122] "RemoveContainer" containerID="3a2622d9487c9b642d43051ef2cc61dc20ecc4c06f59ddf87738b709444f9880" Oct 30 16:42:01.236185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.236160 4861 generic.go:356] "Generic (PLEG): container finished" podID="143c2049-fabb-4d96-8c30-6c11b24b2c4c" containerID="9a4f1dd68dd0f457625add2d202c132570b82072f34cf6d3671b0992b75493ac" exitCode=0 Oct 30 16:42:01.236322 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.236209 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" Oct 30 16:42:01.236322 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.236235 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" event={"ID":"143c2049-fabb-4d96-8c30-6c11b24b2c4c","Type":"ContainerDied","Data":"9a4f1dd68dd0f457625add2d202c132570b82072f34cf6d3671b0992b75493ac"} Oct 30 16:42:01.236322 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.236294 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p" event={"ID":"143c2049-fabb-4d96-8c30-6c11b24b2c4c","Type":"ContainerDied","Data":"800de2288e88275f59395f69f754c17eeb3aee2afe8709ee24ff14ca7ad9f411"} Oct 30 16:42:01.245825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.245805 4861 scope.go:122] "RemoveContainer" containerID="3a2622d9487c9b642d43051ef2cc61dc20ecc4c06f59ddf87738b709444f9880" Oct 30 16:42:01.246095 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:42:01.246077 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a2622d9487c9b642d43051ef2cc61dc20ecc4c06f59ddf87738b709444f9880\": container with ID starting with 3a2622d9487c9b642d43051ef2cc61dc20ecc4c06f59ddf87738b709444f9880 not found: ID does not exist" containerID="3a2622d9487c9b642d43051ef2cc61dc20ecc4c06f59ddf87738b709444f9880" Oct 30 16:42:01.246147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.246103 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a2622d9487c9b642d43051ef2cc61dc20ecc4c06f59ddf87738b709444f9880"} err="failed to get container status \"3a2622d9487c9b642d43051ef2cc61dc20ecc4c06f59ddf87738b709444f9880\": rpc error: code = NotFound desc = could not find container \"3a2622d9487c9b642d43051ef2cc61dc20ecc4c06f59ddf87738b709444f9880\": container with ID starting with 3a2622d9487c9b642d43051ef2cc61dc20ecc4c06f59ddf87738b709444f9880 not found: ID does not exist" Oct 30 16:42:01.246147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.246120 4861 scope.go:122] "RemoveContainer" containerID="9a4f1dd68dd0f457625add2d202c132570b82072f34cf6d3671b0992b75493ac" Oct 30 16:42:01.255294 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.255276 4861 scope.go:122] "RemoveContainer" containerID="9a4f1dd68dd0f457625add2d202c132570b82072f34cf6d3671b0992b75493ac" Oct 30 16:42:01.255593 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:42:01.255567 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4f1dd68dd0f457625add2d202c132570b82072f34cf6d3671b0992b75493ac\": container with ID starting with 9a4f1dd68dd0f457625add2d202c132570b82072f34cf6d3671b0992b75493ac not found: ID does not exist" containerID="9a4f1dd68dd0f457625add2d202c132570b82072f34cf6d3671b0992b75493ac" Oct 30 16:42:01.255694 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.255601 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4f1dd68dd0f457625add2d202c132570b82072f34cf6d3671b0992b75493ac"} err="failed to get container status \"9a4f1dd68dd0f457625add2d202c132570b82072f34cf6d3671b0992b75493ac\": rpc error: code = NotFound desc = could not find container \"9a4f1dd68dd0f457625add2d202c132570b82072f34cf6d3671b0992b75493ac\": container with ID starting with 9a4f1dd68dd0f457625add2d202c132570b82072f34cf6d3671b0992b75493ac not found: ID does not exist" Oct 30 16:42:01.291866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.291843 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p"] Oct 30 16:42:01.297452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.297432 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-83d885-7d5775967-fk48p"] Oct 30 16:42:01.332822 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.332787 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d"] Oct 30 16:42:01.338393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.338370 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-83d885-8669d5b896-dcg9d"] Oct 30 16:42:01.599814 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.599702 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:01.808402 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.808371 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="143c2049-fabb-4d96-8c30-6c11b24b2c4c" path="/var/lib/kubelet/pods/143c2049-fabb-4d96-8c30-6c11b24b2c4c/volumes" Oct 30 16:42:01.808904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:01.808889 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d599d5-d4b9-4708-ba37-beecd9e6cfbe" path="/var/lib/kubelet/pods/29d599d5-d4b9-4708-ba37-beecd9e6cfbe/volumes" Oct 30 16:42:02.689295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.689267 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm"] Oct 30 16:42:02.690008 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.689985 4861 cpu_manager.go:414] "containerMap: removing container" podUID="34c86fa8-9c6e-4aac-92e4-434250b1b3a0" containerName="payload-pre-processing" Oct 30 16:42:02.690008 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.690003 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c86fa8-9c6e-4aac-92e4-434250b1b3a0" containerName="payload-pre-processing" Oct 30 16:42:02.690203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.690034 4861 cpu_manager.go:414] "containerMap: removing container" podUID="143c2049-fabb-4d96-8c30-6c11b24b2c4c" containerName="payload-pre-processing" Oct 30 16:42:02.690203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.690040 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143c2049-fabb-4d96-8c30-6c11b24b2c4c" containerName="payload-pre-processing" Oct 30 16:42:02.690203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.690047 4861 cpu_manager.go:414] "containerMap: removing container" podUID="29d599d5-d4b9-4708-ba37-beecd9e6cfbe" containerName="payload-processing" Oct 30 16:42:02.690203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.690052 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d599d5-d4b9-4708-ba37-beecd9e6cfbe" containerName="payload-processing" Oct 30 16:42:02.690203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.690066 4861 cpu_manager.go:414] "containerMap: removing container" podUID="c44a2b96-5f28-4bee-89b1-3aa1967fdad8" containerName="payload-processing" Oct 30 16:42:02.690203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.690071 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44a2b96-5f28-4bee-89b1-3aa1967fdad8" containerName="payload-processing" Oct 30 16:42:02.690203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.690175 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c86fa8-9c6e-4aac-92e4-434250b1b3a0" containerName="payload-pre-processing" Oct 30 16:42:02.690203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.690189 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="143c2049-fabb-4d96-8c30-6c11b24b2c4c" containerName="payload-pre-processing" Oct 30 16:42:02.690203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.690195 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d599d5-d4b9-4708-ba37-beecd9e6cfbe" containerName="payload-processing" Oct 30 16:42:02.690203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.690202 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44a2b96-5f28-4bee-89b1-3aa1967fdad8" containerName="payload-processing" Oct 30 16:42:02.695148 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.695130 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.723191 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.723171 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-90df54-openshift-default-dockercfg-gv2k6\"" Oct 30 16:42:02.727712 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.727691 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm"] Oct 30 16:42:02.762974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.762940 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-podinfo\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.763144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.763004 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-workload-socket\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.763144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.763063 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-workload-certs\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.763271 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.763191 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istiod-ca-cert\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.763344 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.763319 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-credential-socket\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.763488 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.763471 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-envoy\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.763567 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.763553 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncj8b\" (UniqueName: \"kubernetes.io/projected/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-kube-api-access-ncj8b\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.763655 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.763641 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-data\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.763695 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.763669 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-token\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.865206 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.865181 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-credential-socket\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.865576 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.865439 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-envoy\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.865576 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.865516 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-credential-socket\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.865576 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.865525 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncj8b\" (UniqueName: \"kubernetes.io/projected/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-kube-api-access-ncj8b\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.865900 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.865645 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-data\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.865900 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.865678 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-token\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.865900 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.865777 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-podinfo\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.865900 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.865849 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-workload-socket\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.865900 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.865877 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-workload-certs\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.866161 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.865929 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istiod-ca-cert\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.866161 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.865968 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-data\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.866237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.866160 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-workload-socket\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.866304 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.866283 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-workload-certs\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.866512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.866493 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istiod-ca-cert\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.867879 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.867859 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-envoy\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.868529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.868511 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-podinfo\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.889882 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.889855 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncj8b\" (UniqueName: \"kubernetes.io/projected/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-kube-api-access-ncj8b\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:02.889988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:02.889927 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-token\") pod \"e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:03.008472 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:03.008445 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:03.411253 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:03.411205 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm"] Oct 30 16:42:03.418887 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:03.418466 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:42:03.419118 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:03.419094 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:42:03.419259 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:03.419243 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:42:03.476396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:03.476368 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:03.637545 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:03.637402 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:03.734477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:03.734452 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" Oct 30 16:42:03.821593 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:03.821489 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50d1af4a-8f20-4200-85b7-615c9475d573-kube-api-access-7nhz4\" (UniqueName: \"kubernetes.io/projected/50d1af4a-8f20-4200-85b7-615c9475d573-kube-api-access-7nhz4\") pod \"50d1af4a-8f20-4200-85b7-615c9475d573\" (UID: \"50d1af4a-8f20-4200-85b7-615c9475d573\") " Oct 30 16:42:03.821907 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:03.821883 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50d1af4a-8f20-4200-85b7-615c9475d573-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d1af4a-8f20-4200-85b7-615c9475d573-plugins-config-volume\") pod \"50d1af4a-8f20-4200-85b7-615c9475d573\" (UID: \"50d1af4a-8f20-4200-85b7-615c9475d573\") " Oct 30 16:42:03.824124 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:03.823984 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d1af4a-8f20-4200-85b7-615c9475d573-plugins-config-volume" pod "50d1af4a-8f20-4200-85b7-615c9475d573" (UID: "50d1af4a-8f20-4200-85b7-615c9475d573"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:03.840479 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:03.840450 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d1af4a-8f20-4200-85b7-615c9475d573-kube-api-access-7nhz4" pod "50d1af4a-8f20-4200-85b7-615c9475d573" (UID: "50d1af4a-8f20-4200-85b7-615c9475d573"). InnerVolumeSpecName "kube-api-access-7nhz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:03.925682 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:03.925653 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50d1af4a-8f20-4200-85b7-615c9475d573-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:03.925682 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:03.925685 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nhz4\" (UniqueName: \"kubernetes.io/projected/50d1af4a-8f20-4200-85b7-615c9475d573-kube-api-access-7nhz4\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:03.951274 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:03.951245 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" Oct 30 16:42:04.026350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.026270 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57333841-13b5-466b-8fc5-bf4c0b5ea649-kube-api-access-x5fwl\" (UniqueName: \"kubernetes.io/projected/57333841-13b5-466b-8fc5-bf4c0b5ea649-kube-api-access-x5fwl\") pod \"57333841-13b5-466b-8fc5-bf4c0b5ea649\" (UID: \"57333841-13b5-466b-8fc5-bf4c0b5ea649\") " Oct 30 16:42:04.026521 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.026486 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57333841-13b5-466b-8fc5-bf4c0b5ea649-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57333841-13b5-466b-8fc5-bf4c0b5ea649-plugins-config-volume\") pod \"57333841-13b5-466b-8fc5-bf4c0b5ea649\" (UID: \"57333841-13b5-466b-8fc5-bf4c0b5ea649\") " Oct 30 16:42:04.026902 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.026864 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57333841-13b5-466b-8fc5-bf4c0b5ea649-plugins-config-volume" pod "57333841-13b5-466b-8fc5-bf4c0b5ea649" (UID: "57333841-13b5-466b-8fc5-bf4c0b5ea649"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:04.027171 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.027151 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57333841-13b5-466b-8fc5-bf4c0b5ea649-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:04.028924 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.028895 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57333841-13b5-466b-8fc5-bf4c0b5ea649-kube-api-access-x5fwl" pod "57333841-13b5-466b-8fc5-bf4c0b5ea649" (UID: "57333841-13b5-466b-8fc5-bf4c0b5ea649"). InnerVolumeSpecName "kube-api-access-x5fwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:04.138755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.129754 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5fwl\" (UniqueName: \"kubernetes.io/projected/57333841-13b5-466b-8fc5-bf4c0b5ea649-kube-api-access-x5fwl\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:04.413016 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.412922 4861 generic.go:356] "Generic (PLEG): container finished" podID="50d1af4a-8f20-4200-85b7-615c9475d573" containerID="c375ccc76ffe115d26f321dacd7500c0c4f8eccb5f621f6fda43fd6837287524" exitCode=0 Oct 30 16:42:04.413397 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.413372 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" event={"ID":"50d1af4a-8f20-4200-85b7-615c9475d573","Type":"ContainerDied","Data":"c375ccc76ffe115d26f321dacd7500c0c4f8eccb5f621f6fda43fd6837287524"} Oct 30 16:42:04.413552 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.413537 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" event={"ID":"50d1af4a-8f20-4200-85b7-615c9475d573","Type":"ContainerDied","Data":"91472f2d800994a198fdf253b723e867c5fa01ebf6ba628aa766f26c4babe118"} Oct 30 16:42:04.413675 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.413663 4861 scope.go:122] "RemoveContainer" containerID="c375ccc76ffe115d26f321dacd7500c0c4f8eccb5f621f6fda43fd6837287524" Oct 30 16:42:04.413920 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.413900 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq" Oct 30 16:42:04.422961 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.419520 4861 generic.go:356] "Generic (PLEG): container finished" podID="57333841-13b5-466b-8fc5-bf4c0b5ea649" containerID="3552dbb28daa4e46d3c8dcde4eb8c572563a253df3a8308721968bc21f53460e" exitCode=0 Oct 30 16:42:04.422961 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.419635 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" event={"ID":"57333841-13b5-466b-8fc5-bf4c0b5ea649","Type":"ContainerDied","Data":"3552dbb28daa4e46d3c8dcde4eb8c572563a253df3a8308721968bc21f53460e"} Oct 30 16:42:04.422961 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.419678 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" event={"ID":"57333841-13b5-466b-8fc5-bf4c0b5ea649","Type":"ContainerDied","Data":"f7e84a86e03ac895d0b674bc9e76e790a6dec609ec9ee2a5bf538fef4d9036e3"} Oct 30 16:42:04.422961 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.419780 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s" Oct 30 16:42:04.441929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.441437 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" event={"ID":"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7","Type":"ContainerStarted","Data":"9d93182700ea50160a9b3a9777864a1056d34d5fc205e6d0ee04814948291d81"} Oct 30 16:42:04.441929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.441500 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" event={"ID":"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7","Type":"ContainerStarted","Data":"d1f04cde6953bcc448932d516ba057defa97a2bc06989ae82c4341e782003ff9"} Oct 30 16:42:04.464323 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.464299 4861 scope.go:122] "RemoveContainer" containerID="c375ccc76ffe115d26f321dacd7500c0c4f8eccb5f621f6fda43fd6837287524" Oct 30 16:42:04.464704 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:42:04.464677 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c375ccc76ffe115d26f321dacd7500c0c4f8eccb5f621f6fda43fd6837287524\": container with ID starting with c375ccc76ffe115d26f321dacd7500c0c4f8eccb5f621f6fda43fd6837287524 not found: ID does not exist" containerID="c375ccc76ffe115d26f321dacd7500c0c4f8eccb5f621f6fda43fd6837287524" Oct 30 16:42:04.464790 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.464734 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c375ccc76ffe115d26f321dacd7500c0c4f8eccb5f621f6fda43fd6837287524"} err="failed to get container status \"c375ccc76ffe115d26f321dacd7500c0c4f8eccb5f621f6fda43fd6837287524\": rpc error: code = NotFound desc = could not find container \"c375ccc76ffe115d26f321dacd7500c0c4f8eccb5f621f6fda43fd6837287524\": container with ID starting with c375ccc76ffe115d26f321dacd7500c0c4f8eccb5f621f6fda43fd6837287524 not found: ID does not exist" Oct 30 16:42:04.464790 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.464765 4861 scope.go:122] "RemoveContainer" containerID="3552dbb28daa4e46d3c8dcde4eb8c572563a253df3a8308721968bc21f53460e" Oct 30 16:42:04.475451 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.475428 4861 scope.go:122] "RemoveContainer" containerID="3552dbb28daa4e46d3c8dcde4eb8c572563a253df3a8308721968bc21f53460e" Oct 30 16:42:04.475823 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:42:04.475799 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3552dbb28daa4e46d3c8dcde4eb8c572563a253df3a8308721968bc21f53460e\": container with ID starting with 3552dbb28daa4e46d3c8dcde4eb8c572563a253df3a8308721968bc21f53460e not found: ID does not exist" containerID="3552dbb28daa4e46d3c8dcde4eb8c572563a253df3a8308721968bc21f53460e" Oct 30 16:42:04.475897 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.475833 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3552dbb28daa4e46d3c8dcde4eb8c572563a253df3a8308721968bc21f53460e"} err="failed to get container status \"3552dbb28daa4e46d3c8dcde4eb8c572563a253df3a8308721968bc21f53460e\": rpc error: code = NotFound desc = could not find container \"3552dbb28daa4e46d3c8dcde4eb8c572563a253df3a8308721968bc21f53460e\": container with ID starting with 3552dbb28daa4e46d3c8dcde4eb8c572563a253df3a8308721968bc21f53460e not found: ID does not exist" Oct 30 16:42:04.535770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.535693 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" podStartSLOduration=2.5356796150000003 podStartE2EDuration="2.535679615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:04.532973259 +0000 UTC m=+1154.875647301" watchObservedRunningTime="2026-10-30 16:42:04.535679615 +0000 UTC m=+1154.878353653" Oct 30 16:42:04.579800 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.579717 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq"] Oct 30 16:42:04.583487 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.583464 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-82381960-579cc97966-6k7zq"] Oct 30 16:42:04.589852 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.589826 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:04.686232 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.686140 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s"] Oct 30 16:42:04.723387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.723354 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-82381960-6fcc9df86c-jl94s"] Oct 30 16:42:04.874659 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:04.874626 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:05.018749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:05.015058 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:05.019299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:05.018801 4861 patch_prober.go:29] interesting pod/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:42:05.019299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:05.018869 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" podUID="66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 16:42:05.822745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:05.820157 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50d1af4a-8f20-4200-85b7-615c9475d573" path="/var/lib/kubelet/pods/50d1af4a-8f20-4200-85b7-615c9475d573/volumes" Oct 30 16:42:05.822745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:05.820817 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57333841-13b5-466b-8fc5-bf4c0b5ea649" path="/var/lib/kubelet/pods/57333841-13b5-466b-8fc5-bf4c0b5ea649/volumes" Oct 30 16:42:06.018751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:06.017031 4861 patch_prober.go:29] interesting pod/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:42:06.018751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:06.017090 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" podUID="66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 16:42:06.152195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:06.152113 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:06.861463 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:06.861434 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:07.011089 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:07.011052 4861 patch_prober.go:29] interesting pod/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:42:07.011367 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:07.011337 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" podUID="66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 16:42:08.014748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:08.009613 4861 patch_prober.go:29] interesting pod/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:42:08.014748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:08.009673 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" podUID="66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 16:42:08.197510 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:08.197466 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:09.017748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:09.012007 4861 patch_prober.go:29] interesting pod/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:42:09.017748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:09.012062 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" podUID="66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 16:42:10.011400 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:10.011363 4861 patch_prober.go:29] interesting pod/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 16:42:10.011683 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:10.011642 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" podUID="66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 16:42:10.336141 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:10.336054 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:10.915856 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:10.915799 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:11.013666 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:11.013632 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:11.014063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:11.014044 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:11.014620 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:11.014599 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:11.056067 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:11.056040 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:11.755076 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:11.755028 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:11.905525 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:11.905489 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:12.716817 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:12.716517 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:14.456155 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:14.456114 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:16.255279 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:16.255250 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:16.410750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:16.405772 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:16.588457 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:16.588375 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:17.897742 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:17.897692 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv"] Oct 30 16:42:17.898223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:17.897971 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" podUID="51859589-1590-41e5-a1be-73680e4ced59" containerName="maas-api" containerID="cri-o://720c1a0fc81ebc57506cd25d818961fd935566e28d9ff35f41720f8f1169c168" gracePeriod=30 Oct 30 16:42:17.984420 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:17.984386 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:18.342661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.342619 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:42:18.487411 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.487320 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51859589-1590-41e5-a1be-73680e4ced59-kube-api-access-zgdz6\" (UniqueName: \"kubernetes.io/projected/51859589-1590-41e5-a1be-73680e4ced59-kube-api-access-zgdz6\") pod \"51859589-1590-41e5-a1be-73680e4ced59\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " Oct 30 16:42:18.487783 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.487757 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-metrics-certs\") pod \"51859589-1590-41e5-a1be-73680e4ced59\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " Oct 30 16:42:18.488070 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.488047 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51859589-1590-41e5-a1be-73680e4ced59-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51859589-1590-41e5-a1be-73680e4ced59-openshift-service-ca\") pod \"51859589-1590-41e5-a1be-73680e4ced59\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " Oct 30 16:42:18.503747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.488397 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-maas-api-tls\") pod \"51859589-1590-41e5-a1be-73680e4ced59\" (UID: \"51859589-1590-41e5-a1be-73680e4ced59\") " Oct 30 16:42:18.507897 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.489041 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51859589-1590-41e5-a1be-73680e4ced59-openshift-service-ca" pod "51859589-1590-41e5-a1be-73680e4ced59" (UID: "51859589-1590-41e5-a1be-73680e4ced59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:18.514781 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.511339 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:18.522178 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.522078 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51859589-1590-41e5-a1be-73680e4ced59-kube-api-access-zgdz6" pod "51859589-1590-41e5-a1be-73680e4ced59" (UID: "51859589-1590-41e5-a1be-73680e4ced59"). InnerVolumeSpecName "kube-api-access-zgdz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:18.557231 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.557148 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-maas-api-tls" pod "51859589-1590-41e5-a1be-73680e4ced59" (UID: "51859589-1590-41e5-a1be-73680e4ced59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:18.559049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.557507 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-metrics-certs" pod "51859589-1590-41e5-a1be-73680e4ced59" (UID: "51859589-1590-41e5-a1be-73680e4ced59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:18.656147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.655564 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:18.656147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.655598 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgdz6\" (UniqueName: \"kubernetes.io/projected/51859589-1590-41e5-a1be-73680e4ced59-kube-api-access-zgdz6\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:18.656147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.655611 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51859589-1590-41e5-a1be-73680e4ced59-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:18.656147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.655623 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51859589-1590-41e5-a1be-73680e4ced59-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:18.691653 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.691493 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:18.701395 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.700510 4861 generic.go:356] "Generic (PLEG): container finished" podID="51859589-1590-41e5-a1be-73680e4ced59" containerID="720c1a0fc81ebc57506cd25d818961fd935566e28d9ff35f41720f8f1169c168" exitCode=0 Oct 30 16:42:18.701395 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.700571 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" Oct 30 16:42:18.701395 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.700902 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" event={"ID":"51859589-1590-41e5-a1be-73680e4ced59","Type":"ContainerDied","Data":"720c1a0fc81ebc57506cd25d818961fd935566e28d9ff35f41720f8f1169c168"} Oct 30 16:42:18.701395 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.701352 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv" event={"ID":"51859589-1590-41e5-a1be-73680e4ced59","Type":"ContainerDied","Data":"4b56d318bb9260b796b588fcb497d90beb04c1793d2843ab4cbb00a77c152acc"} Oct 30 16:42:18.701395 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.701402 4861 scope.go:122] "RemoveContainer" containerID="720c1a0fc81ebc57506cd25d818961fd935566e28d9ff35f41720f8f1169c168" Oct 30 16:42:18.728065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.725870 4861 scope.go:122] "RemoveContainer" containerID="720c1a0fc81ebc57506cd25d818961fd935566e28d9ff35f41720f8f1169c168" Oct 30 16:42:18.728065 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:42:18.726299 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720c1a0fc81ebc57506cd25d818961fd935566e28d9ff35f41720f8f1169c168\": container with ID starting with 720c1a0fc81ebc57506cd25d818961fd935566e28d9ff35f41720f8f1169c168 not found: ID does not exist" containerID="720c1a0fc81ebc57506cd25d818961fd935566e28d9ff35f41720f8f1169c168" Oct 30 16:42:18.728065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.726332 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720c1a0fc81ebc57506cd25d818961fd935566e28d9ff35f41720f8f1169c168"} err="failed to get container status \"720c1a0fc81ebc57506cd25d818961fd935566e28d9ff35f41720f8f1169c168\": rpc error: code = NotFound desc = could not find container \"720c1a0fc81ebc57506cd25d818961fd935566e28d9ff35f41720f8f1169c168\": container with ID starting with 720c1a0fc81ebc57506cd25d818961fd935566e28d9ff35f41720f8f1169c168 not found: ID does not exist" Oct 30 16:42:18.792837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.784873 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv"] Oct 30 16:42:18.795642 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.794522 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-83d885-86549d5df4-vqtgv"] Oct 30 16:42:18.897782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:18.897151 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:19.117981 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:19.117870 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:19.261060 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:19.260919 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:19.462805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:19.457145 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:19.627749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:19.627138 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:19.778745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:19.778626 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:19.818744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:19.818022 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51859589-1590-41e5-a1be-73680e4ced59" path="/var/lib/kubelet/pods/51859589-1590-41e5-a1be-73680e4ced59/volumes" Oct 30 16:42:19.935641 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:19.931953 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:20.276036 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:20.275991 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:20.290776 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:42:20.290745 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:42:21.809880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:21.809841 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:23.835635 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:23.835606 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:25.716313 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:25.716277 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:26.141486 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:26.141453 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:26.617476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:26.617450 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:26.759984 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:26.759953 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:26.990002 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:26.989975 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:27.761827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:27.759467 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:27.899678 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:27.899629 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:28.073678 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:28.073592 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm"] Oct 30 16:42:28.074317 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:28.074285 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" podUID="66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" containerName="istio-proxy" containerID="cri-o://9d93182700ea50160a9b3a9777864a1056d34d5fc205e6d0ee04814948291d81" gracePeriod=30 Oct 30 16:42:28.105297 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:28.105235 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:29.883860 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:29.883829 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:31.883771 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:31.883690 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:33.301772 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.301750 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:33.304941 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.304918 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:33.415286 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.415218 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-credential-socket\") pod \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " Oct 30 16:42:33.415527 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.415361 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-workload-certs\") pod \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " Oct 30 16:42:33.415527 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.415393 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istiod-ca-cert\") pod \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " Oct 30 16:42:33.415527 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.415414 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-envoy\") pod \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " Oct 30 16:42:33.415527 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.415452 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-podinfo\") pod \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " Oct 30 16:42:33.415527 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.415482 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-token\" (UniqueName: \"kubernetes.io/projected/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-token\") pod \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " Oct 30 16:42:33.418409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.415521 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-credential-socket" pod "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" (UID: "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:33.418409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.415592 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-kube-api-access-ncj8b\" (UniqueName: \"kubernetes.io/projected/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-kube-api-access-ncj8b\") pod \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " Oct 30 16:42:33.418409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.415630 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-data\") pod \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " Oct 30 16:42:33.418409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.415681 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-workload-socket\") pod \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\" (UID: \"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7\") " Oct 30 16:42:33.418409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.415708 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-workload-certs" pod "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" (UID: "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:33.418409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.415961 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-data" pod "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" (UID: "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:33.418409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.416142 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-workload-socket" pod "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" (UID: "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:33.418409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.416211 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istiod-ca-cert" pod "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" (UID: "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:33.418409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.416389 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:33.418409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.416409 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:33.418409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.416419 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:33.418409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.416426 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:33.418409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.416434 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:33.419167 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.419116 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-podinfo" pod "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" (UID: "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:42:33.419167 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.419138 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-envoy" pod "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" (UID: "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:33.419258 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.419208 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-token" pod "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" (UID: "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:33.419258 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.419233 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-kube-api-access-ncj8b" pod "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" (UID: "66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7"). InnerVolumeSpecName "kube-api-access-ncj8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:33.517541 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.517514 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:33.517541 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.517541 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:33.517709 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.517550 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:33.517709 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.517559 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncj8b\" (UniqueName: \"kubernetes.io/projected/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7-kube-api-access-ncj8b\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:33.920754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.917711 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:33.938139 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.936704 4861 generic.go:356] "Generic (PLEG): container finished" podID="66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" containerID="9d93182700ea50160a9b3a9777864a1056d34d5fc205e6d0ee04814948291d81" exitCode=0 Oct 30 16:42:33.938139 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.936904 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" event={"ID":"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7","Type":"ContainerDied","Data":"9d93182700ea50160a9b3a9777864a1056d34d5fc205e6d0ee04814948291d81"} Oct 30 16:42:33.938139 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.936958 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" event={"ID":"66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7","Type":"ContainerDied","Data":"d1f04cde6953bcc448932d516ba057defa97a2bc06989ae82c4341e782003ff9"} Oct 30 16:42:33.938139 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.936990 4861 scope.go:122] "RemoveContainer" containerID="9d93182700ea50160a9b3a9777864a1056d34d5fc205e6d0ee04814948291d81" Oct 30 16:42:33.938139 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.937165 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm" Oct 30 16:42:33.980922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.980636 4861 scope.go:122] "RemoveContainer" containerID="9d93182700ea50160a9b3a9777864a1056d34d5fc205e6d0ee04814948291d81" Oct 30 16:42:33.981655 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:42:33.981613 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d93182700ea50160a9b3a9777864a1056d34d5fc205e6d0ee04814948291d81\": container with ID starting with 9d93182700ea50160a9b3a9777864a1056d34d5fc205e6d0ee04814948291d81 not found: ID does not exist" containerID="9d93182700ea50160a9b3a9777864a1056d34d5fc205e6d0ee04814948291d81" Oct 30 16:42:33.982070 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:33.982037 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d93182700ea50160a9b3a9777864a1056d34d5fc205e6d0ee04814948291d81"} err="failed to get container status \"9d93182700ea50160a9b3a9777864a1056d34d5fc205e6d0ee04814948291d81\": rpc error: code = NotFound desc = could not find container \"9d93182700ea50160a9b3a9777864a1056d34d5fc205e6d0ee04814948291d81\": container with ID starting with 9d93182700ea50160a9b3a9777864a1056d34d5fc205e6d0ee04814948291d81 not found: ID does not exist" Oct 30 16:42:34.010986 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:34.010956 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm"] Oct 30 16:42:34.015452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:34.015419 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-90df54-openshift-default-5758856cf9-pnfnm"] Oct 30 16:42:35.767475 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:35.767444 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:35.839105 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:35.836617 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" path="/var/lib/kubelet/pods/66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7/volumes" Oct 30 16:42:37.188051 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:37.188020 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:37.585769 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:37.585709 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:39.382230 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:39.382194 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:41.191834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:41.191801 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:42.881419 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:42.881385 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:43.105113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:43.105084 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:44.888038 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:44.887996 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:46.350300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:46.350268 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w"] Oct 30 16:42:46.350951 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:46.350554 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" podUID="051f1a45-22fb-410a-93a8-d43cd5c730f1" containerName="maas-api" containerID="cri-o://8b1bd4f5437947ab6140c52c881acacf7c2231e40ac325fb7a78270dfb9123b2" gracePeriod=30 Oct 30 16:42:46.445100 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:46.445065 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:46.790537 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:46.790497 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:42:46.899500 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:46.899465 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051f1a45-22fb-410a-93a8-d43cd5c730f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051f1a45-22fb-410a-93a8-d43cd5c730f1-openshift-service-ca\") pod \"051f1a45-22fb-410a-93a8-d43cd5c730f1\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " Oct 30 16:42:46.899690 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:46.899527 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051f1a45-22fb-410a-93a8-d43cd5c730f1-kube-api-access-xxxff\" (UniqueName: \"kubernetes.io/projected/051f1a45-22fb-410a-93a8-d43cd5c730f1-kube-api-access-xxxff\") pod \"051f1a45-22fb-410a-93a8-d43cd5c730f1\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " Oct 30 16:42:46.899690 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:46.899632 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-metrics-certs\") pod \"051f1a45-22fb-410a-93a8-d43cd5c730f1\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " Oct 30 16:42:46.899857 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:46.899843 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-maas-api-tls\") pod \"051f1a45-22fb-410a-93a8-d43cd5c730f1\" (UID: \"051f1a45-22fb-410a-93a8-d43cd5c730f1\") " Oct 30 16:42:46.899915 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:46.899857 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051f1a45-22fb-410a-93a8-d43cd5c730f1-openshift-service-ca" pod "051f1a45-22fb-410a-93a8-d43cd5c730f1" (UID: "051f1a45-22fb-410a-93a8-d43cd5c730f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:46.900478 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:46.900455 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/051f1a45-22fb-410a-93a8-d43cd5c730f1-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:46.902414 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:46.902385 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051f1a45-22fb-410a-93a8-d43cd5c730f1-kube-api-access-xxxff" pod "051f1a45-22fb-410a-93a8-d43cd5c730f1" (UID: "051f1a45-22fb-410a-93a8-d43cd5c730f1"). InnerVolumeSpecName "kube-api-access-xxxff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:46.902414 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:46.902403 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-maas-api-tls" pod "051f1a45-22fb-410a-93a8-d43cd5c730f1" (UID: "051f1a45-22fb-410a-93a8-d43cd5c730f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:46.902601 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:46.902415 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-metrics-certs" pod "051f1a45-22fb-410a-93a8-d43cd5c730f1" (UID: "051f1a45-22fb-410a-93a8-d43cd5c730f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:47.004448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.004036 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:47.004448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.004074 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxxff\" (UniqueName: \"kubernetes.io/projected/051f1a45-22fb-410a-93a8-d43cd5c730f1-kube-api-access-xxxff\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:47.004448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.004089 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/051f1a45-22fb-410a-93a8-d43cd5c730f1-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:42:47.047071 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.046985 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:47.076745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.073397 4861 generic.go:356] "Generic (PLEG): container finished" podID="051f1a45-22fb-410a-93a8-d43cd5c730f1" containerID="8b1bd4f5437947ab6140c52c881acacf7c2231e40ac325fb7a78270dfb9123b2" exitCode=0 Oct 30 16:42:47.076745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.073444 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" event={"ID":"051f1a45-22fb-410a-93a8-d43cd5c730f1","Type":"ContainerDied","Data":"8b1bd4f5437947ab6140c52c881acacf7c2231e40ac325fb7a78270dfb9123b2"} Oct 30 16:42:47.076745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.073491 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" event={"ID":"051f1a45-22fb-410a-93a8-d43cd5c730f1","Type":"ContainerDied","Data":"a4aa46d6d0beb7b50572cd842a6ba460bf1ff95da8a3553ccbb5f58d074fb398"} Oct 30 16:42:47.076745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.073518 4861 scope.go:122] "RemoveContainer" containerID="8b1bd4f5437947ab6140c52c881acacf7c2231e40ac325fb7a78270dfb9123b2" Oct 30 16:42:47.076745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.073680 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" Oct 30 16:42:47.155831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.152600 4861 scope.go:122] "RemoveContainer" containerID="8b1bd4f5437947ab6140c52c881acacf7c2231e40ac325fb7a78270dfb9123b2" Oct 30 16:42:47.155831 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:42:47.153090 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1bd4f5437947ab6140c52c881acacf7c2231e40ac325fb7a78270dfb9123b2\": container with ID starting with 8b1bd4f5437947ab6140c52c881acacf7c2231e40ac325fb7a78270dfb9123b2 not found: ID does not exist" containerID="8b1bd4f5437947ab6140c52c881acacf7c2231e40ac325fb7a78270dfb9123b2" Oct 30 16:42:47.155831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.153124 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1bd4f5437947ab6140c52c881acacf7c2231e40ac325fb7a78270dfb9123b2"} err="failed to get container status \"8b1bd4f5437947ab6140c52c881acacf7c2231e40ac325fb7a78270dfb9123b2\": rpc error: code = NotFound desc = could not find container \"8b1bd4f5437947ab6140c52c881acacf7c2231e40ac325fb7a78270dfb9123b2\": container with ID starting with 8b1bd4f5437947ab6140c52c881acacf7c2231e40ac325fb7a78270dfb9123b2 not found: ID does not exist" Oct 30 16:42:47.190627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.190469 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:47.198879 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.198851 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w"] Oct 30 16:42:47.203048 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.203021 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w"] Oct 30 16:42:47.349257 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.349149 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:47.759588 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.759538 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:47.808201 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:47.808175 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051f1a45-22fb-410a-93a8-d43cd5c730f1" path="/var/lib/kubelet/pods/051f1a45-22fb-410a-93a8-d43cd5c730f1/volumes" Oct 30 16:42:49.313287 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:49.313254 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:49.769391 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:49.769355 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7410c7-846774d6dc-v278w" podUID="051f1a45-22fb-410a-93a8-d43cd5c730f1" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.101:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:49.987489 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:49.987462 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:42:49.989880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:49.989862 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:42:50.290302 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:42:50.290267 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:42:51.184558 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:51.184525 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:53.180600 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:53.180566 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:54.519319 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:54.519284 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:54.659239 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:54.659201 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:54.858695 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:54.858616 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:55.169425 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:55.169336 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:57.594751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:57.592168 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:59.536245 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:59.536211 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:59.717854 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:59.717620 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:42:59.860275 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:42:59.860191 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:01.794802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:01.794716 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:03.637644 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:03.636364 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:05.191380 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:05.191353 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:05.900498 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:05.900453 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:06.044623 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:06.044595 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:06.125756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:06.124498 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w"] Oct 30 16:43:06.202751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:06.202586 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:06.405329 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:06.405232 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:06.571500 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:06.571464 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:08.284415 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:08.284381 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:10.191239 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:10.191198 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:11.126372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.126330 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" podUID="a8f9abdc-9f33-4065-abac-a3fbf8ec8504" containerName="payload-pre-processing" containerID="cri-o://b9eadfd8f7e7967deeb599d9d8012f273f1585cb6cff12a7785a51bd622ed677" gracePeriod=25 Oct 30 16:43:11.408554 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.408529 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" Oct 30 16:43:11.437641 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.437611 4861 generic.go:356] "Generic (PLEG): container finished" podID="a8f9abdc-9f33-4065-abac-a3fbf8ec8504" containerID="b9eadfd8f7e7967deeb599d9d8012f273f1585cb6cff12a7785a51bd622ed677" exitCode=0 Oct 30 16:43:11.437838 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.437669 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" Oct 30 16:43:11.437838 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.437709 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" event={"ID":"a8f9abdc-9f33-4065-abac-a3fbf8ec8504","Type":"ContainerDied","Data":"b9eadfd8f7e7967deeb599d9d8012f273f1585cb6cff12a7785a51bd622ed677"} Oct 30 16:43:11.437838 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.437809 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w" event={"ID":"a8f9abdc-9f33-4065-abac-a3fbf8ec8504","Type":"ContainerDied","Data":"838381f1fc27a19e55d76fb7b6f1c8d3e7fcb7970d8d64db8eec1145519562d9"} Oct 30 16:43:11.437986 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.437842 4861 scope.go:122] "RemoveContainer" containerID="b9eadfd8f7e7967deeb599d9d8012f273f1585cb6cff12a7785a51bd622ed677" Oct 30 16:43:11.453650 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.453603 4861 scope.go:122] "RemoveContainer" containerID="b9eadfd8f7e7967deeb599d9d8012f273f1585cb6cff12a7785a51bd622ed677" Oct 30 16:43:11.454065 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:43:11.454039 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9eadfd8f7e7967deeb599d9d8012f273f1585cb6cff12a7785a51bd622ed677\": container with ID starting with b9eadfd8f7e7967deeb599d9d8012f273f1585cb6cff12a7785a51bd622ed677 not found: ID does not exist" containerID="b9eadfd8f7e7967deeb599d9d8012f273f1585cb6cff12a7785a51bd622ed677" Oct 30 16:43:11.454188 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.454073 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9eadfd8f7e7967deeb599d9d8012f273f1585cb6cff12a7785a51bd622ed677"} err="failed to get container status \"b9eadfd8f7e7967deeb599d9d8012f273f1585cb6cff12a7785a51bd622ed677\": rpc error: code = NotFound desc = could not find container \"b9eadfd8f7e7967deeb599d9d8012f273f1585cb6cff12a7785a51bd622ed677\": container with ID starting with b9eadfd8f7e7967deeb599d9d8012f273f1585cb6cff12a7785a51bd622ed677 not found: ID does not exist" Oct 30 16:43:11.487975 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.487946 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f9abdc-9f33-4065-abac-a3fbf8ec8504-kube-api-access-txx5m\" (UniqueName: \"kubernetes.io/projected/a8f9abdc-9f33-4065-abac-a3fbf8ec8504-kube-api-access-txx5m\") pod \"a8f9abdc-9f33-4065-abac-a3fbf8ec8504\" (UID: \"a8f9abdc-9f33-4065-abac-a3fbf8ec8504\") " Oct 30 16:43:11.488162 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.488128 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8f9abdc-9f33-4065-abac-a3fbf8ec8504-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f9abdc-9f33-4065-abac-a3fbf8ec8504-plugins-config-volume\") pod \"a8f9abdc-9f33-4065-abac-a3fbf8ec8504\" (UID: \"a8f9abdc-9f33-4065-abac-a3fbf8ec8504\") " Oct 30 16:43:11.488594 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.488568 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f9abdc-9f33-4065-abac-a3fbf8ec8504-plugins-config-volume" pod "a8f9abdc-9f33-4065-abac-a3fbf8ec8504" (UID: "a8f9abdc-9f33-4065-abac-a3fbf8ec8504"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:11.488694 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.488678 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8f9abdc-9f33-4065-abac-a3fbf8ec8504-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:11.490857 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.490827 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f9abdc-9f33-4065-abac-a3fbf8ec8504-kube-api-access-txx5m" pod "a8f9abdc-9f33-4065-abac-a3fbf8ec8504" (UID: "a8f9abdc-9f33-4065-abac-a3fbf8ec8504"). InnerVolumeSpecName "kube-api-access-txx5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:11.493028 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.493001 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs"] Oct 30 16:43:11.590008 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.589978 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txx5m\" (UniqueName: \"kubernetes.io/projected/a8f9abdc-9f33-4065-abac-a3fbf8ec8504-kube-api-access-txx5m\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:11.603756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.603675 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:11.785527 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.785496 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w"] Oct 30 16:43:11.798372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.798347 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1189d4-f6995cdf5-jbd6w"] Oct 30 16:43:11.807497 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.807472 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8f9abdc-9f33-4065-abac-a3fbf8ec8504" path="/var/lib/kubelet/pods/a8f9abdc-9f33-4065-abac-a3fbf8ec8504/volumes" Oct 30 16:43:11.933718 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:11.933689 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:13.706700 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:13.706637 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:15.510958 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:15.510925 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:17.387991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:17.387957 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:19.185020 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:19.184990 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:20.289273 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:43:20.289230 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:43:20.999565 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:20.999536 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:22.992730 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:22.992682 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:24.408289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:24.408245 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm"] Oct 30 16:43:24.408865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:24.408526 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" podUID="5c265c02-98e3-4f69-b447-0811f146b548" containerName="maas-api" containerID="cri-o://fb343a08804294fc5013b6814c4c0747c92c03aa0549433a2cf122311826c7ba" gracePeriod=30 Oct 30 16:43:24.503834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:24.503795 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:24.540705 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:24.540677 4861 generic.go:356] "Generic (PLEG): container finished" podID="5c265c02-98e3-4f69-b447-0811f146b548" containerID="fb343a08804294fc5013b6814c4c0747c92c03aa0549433a2cf122311826c7ba" exitCode=0 Oct 30 16:43:24.540851 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:24.540770 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" event={"ID":"5c265c02-98e3-4f69-b447-0811f146b548","Type":"ContainerDied","Data":"fb343a08804294fc5013b6814c4c0747c92c03aa0549433a2cf122311826c7ba"} Oct 30 16:43:24.856759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:24.854972 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:43:25.096799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.093269 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-maas-api-tls\") pod \"5c265c02-98e3-4f69-b447-0811f146b548\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " Oct 30 16:43:25.098084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.097065 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-metrics-certs\") pod \"5c265c02-98e3-4f69-b447-0811f146b548\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " Oct 30 16:43:25.098512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.098491 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c265c02-98e3-4f69-b447-0811f146b548-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c265c02-98e3-4f69-b447-0811f146b548-openshift-service-ca\") pod \"5c265c02-98e3-4f69-b447-0811f146b548\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " Oct 30 16:43:25.099124 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.099021 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c265c02-98e3-4f69-b447-0811f146b548-openshift-service-ca" pod "5c265c02-98e3-4f69-b447-0811f146b548" (UID: "5c265c02-98e3-4f69-b447-0811f146b548"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:25.099471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.099441 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c265c02-98e3-4f69-b447-0811f146b548-kube-api-access-w79cc\" (UniqueName: \"kubernetes.io/projected/5c265c02-98e3-4f69-b447-0811f146b548-kube-api-access-w79cc\") pod \"5c265c02-98e3-4f69-b447-0811f146b548\" (UID: \"5c265c02-98e3-4f69-b447-0811f146b548\") " Oct 30 16:43:25.151433 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.150865 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-maas-api-tls" pod "5c265c02-98e3-4f69-b447-0811f146b548" (UID: "5c265c02-98e3-4f69-b447-0811f146b548"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:43:25.153075 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.151776 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:25.153075 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.151820 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c265c02-98e3-4f69-b447-0811f146b548-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:25.153075 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.151996 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c265c02-98e3-4f69-b447-0811f146b548-kube-api-access-w79cc" pod "5c265c02-98e3-4f69-b447-0811f146b548" (UID: "5c265c02-98e3-4f69-b447-0811f146b548"). InnerVolumeSpecName "kube-api-access-w79cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:25.159934 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.159905 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-metrics-certs" pod "5c265c02-98e3-4f69-b447-0811f146b548" (UID: "5c265c02-98e3-4f69-b447-0811f146b548"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:43:25.215749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.212246 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:25.252809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.252775 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c265c02-98e3-4f69-b447-0811f146b548-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:25.252809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.252803 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w79cc\" (UniqueName: \"kubernetes.io/projected/5c265c02-98e3-4f69-b447-0811f146b548-kube-api-access-w79cc\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:25.355284 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.355204 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:25.519752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.518603 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:25.561979 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.561949 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" Oct 30 16:43:25.562144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.561975 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm" event={"ID":"5c265c02-98e3-4f69-b447-0811f146b548","Type":"ContainerDied","Data":"3ceff93499f6b350f17346417469e7a01d74651efef40d5998905ee07884800b"} Oct 30 16:43:25.562144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.562052 4861 scope.go:122] "RemoveContainer" containerID="fb343a08804294fc5013b6814c4c0747c92c03aa0549433a2cf122311826c7ba" Oct 30 16:43:25.617118 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.617048 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm"] Oct 30 16:43:25.629332 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.629306 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1189d4-868d76874f-j2jkm"] Oct 30 16:43:25.808127 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.808096 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c265c02-98e3-4f69-b447-0811f146b548" path="/var/lib/kubelet/pods/5c265c02-98e3-4f69-b447-0811f146b548/volumes" Oct 30 16:43:25.860439 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:25.860412 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:26.524666 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.524631 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" podUID="60ec445f-a902-4ed5-b917-a8460f8e4179" containerName="main" containerID="cri-o://6ca8590df2e3d3accd932186bb18d79ba813f83c3fc5ce9ad42eb8472b1bf5a9" gracePeriod=45 Oct 30 16:43:26.799049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.799027 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:43:26.878302 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.878267 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60ec445f-a902-4ed5-b917-a8460f8e4179-tls-certs\" (UniqueName: \"kubernetes.io/secret/60ec445f-a902-4ed5-b917-a8460f8e4179-tls-certs\") pod \"60ec445f-a902-4ed5-b917-a8460f8e4179\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " Oct 30 16:43:26.878517 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.878370 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-tmp-dir\") pod \"60ec445f-a902-4ed5-b917-a8460f8e4179\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " Oct 30 16:43:26.878517 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.878426 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ec445f-a902-4ed5-b917-a8460f8e4179-kube-api-access-bhxsb\" (UniqueName: \"kubernetes.io/projected/60ec445f-a902-4ed5-b917-a8460f8e4179-kube-api-access-bhxsb\") pod \"60ec445f-a902-4ed5-b917-a8460f8e4179\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " Oct 30 16:43:26.878630 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.878546 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-model-cache\") pod \"60ec445f-a902-4ed5-b917-a8460f8e4179\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " Oct 30 16:43:26.878697 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.878681 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-dshm\") pod \"60ec445f-a902-4ed5-b917-a8460f8e4179\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " Oct 30 16:43:26.878796 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.878708 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-tmp-dir" pod "60ec445f-a902-4ed5-b917-a8460f8e4179" (UID: "60ec445f-a902-4ed5-b917-a8460f8e4179"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:26.878796 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.878758 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-home\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-home\") pod \"60ec445f-a902-4ed5-b917-a8460f8e4179\" (UID: \"60ec445f-a902-4ed5-b917-a8460f8e4179\") " Oct 30 16:43:26.879014 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.878977 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-model-cache" pod "60ec445f-a902-4ed5-b917-a8460f8e4179" (UID: "60ec445f-a902-4ed5-b917-a8460f8e4179"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:26.879100 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.879011 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-home" pod "60ec445f-a902-4ed5-b917-a8460f8e4179" (UID: "60ec445f-a902-4ed5-b917-a8460f8e4179"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:26.879444 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.879419 4861 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:26.879444 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.879442 4861 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:26.879444 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.879450 4861 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:26.881350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.881324 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ec445f-a902-4ed5-b917-a8460f8e4179-tls-certs" pod "60ec445f-a902-4ed5-b917-a8460f8e4179" (UID: "60ec445f-a902-4ed5-b917-a8460f8e4179"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:43:26.881350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.881321 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-dshm" pod "60ec445f-a902-4ed5-b917-a8460f8e4179" (UID: "60ec445f-a902-4ed5-b917-a8460f8e4179"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:26.881874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.881849 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ec445f-a902-4ed5-b917-a8460f8e4179-kube-api-access-bhxsb" pod "60ec445f-a902-4ed5-b917-a8460f8e4179" (UID: "60ec445f-a902-4ed5-b917-a8460f8e4179"). InnerVolumeSpecName "kube-api-access-bhxsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:26.981352 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.981323 4861 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60ec445f-a902-4ed5-b917-a8460f8e4179-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:26.981352 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.981347 4861 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60ec445f-a902-4ed5-b917-a8460f8e4179-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:26.981352 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:26.981357 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhxsb\" (UniqueName: \"kubernetes.io/projected/60ec445f-a902-4ed5-b917-a8460f8e4179-kube-api-access-bhxsb\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:27.147944 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:27.147854 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:27.458310 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:27.458231 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:27.578214 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:27.578179 4861 generic.go:356] "Generic (PLEG): container finished" podID="60ec445f-a902-4ed5-b917-a8460f8e4179" containerID="6ca8590df2e3d3accd932186bb18d79ba813f83c3fc5ce9ad42eb8472b1bf5a9" exitCode=0 Oct 30 16:43:27.578214 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:27.578249 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" Oct 30 16:43:27.580425 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:27.578268 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" event={"ID":"60ec445f-a902-4ed5-b917-a8460f8e4179","Type":"ContainerDied","Data":"6ca8590df2e3d3accd932186bb18d79ba813f83c3fc5ce9ad42eb8472b1bf5a9"} Oct 30 16:43:27.580425 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:27.578324 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs" event={"ID":"60ec445f-a902-4ed5-b917-a8460f8e4179","Type":"ContainerDied","Data":"6ec5b57f7205744f4d499c03450cca89ccadd051a49a0c5fa07c939b6cb4f349"} Oct 30 16:43:27.580425 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:27.578352 4861 scope.go:122] "RemoveContainer" containerID="6ca8590df2e3d3accd932186bb18d79ba813f83c3fc5ce9ad42eb8472b1bf5a9" Oct 30 16:43:27.591770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:27.591706 4861 scope.go:122] "RemoveContainer" containerID="6ca8590df2e3d3accd932186bb18d79ba813f83c3fc5ce9ad42eb8472b1bf5a9" Oct 30 16:43:27.592018 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:43:27.591996 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca8590df2e3d3accd932186bb18d79ba813f83c3fc5ce9ad42eb8472b1bf5a9\": container with ID starting with 6ca8590df2e3d3accd932186bb18d79ba813f83c3fc5ce9ad42eb8472b1bf5a9 not found: ID does not exist" containerID="6ca8590df2e3d3accd932186bb18d79ba813f83c3fc5ce9ad42eb8472b1bf5a9" Oct 30 16:43:27.592107 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:27.592035 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca8590df2e3d3accd932186bb18d79ba813f83c3fc5ce9ad42eb8472b1bf5a9"} err="failed to get container status \"6ca8590df2e3d3accd932186bb18d79ba813f83c3fc5ce9ad42eb8472b1bf5a9\": rpc error: code = NotFound desc = could not find container \"6ca8590df2e3d3accd932186bb18d79ba813f83c3fc5ce9ad42eb8472b1bf5a9\": container with ID starting with 6ca8590df2e3d3accd932186bb18d79ba813f83c3fc5ce9ad42eb8472b1bf5a9 not found: ID does not exist" Oct 30 16:43:27.613101 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:27.613075 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:27.734747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:27.732831 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs"] Oct 30 16:43:27.754751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:27.753521 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-82381960/e2e-lifecycle-model-82381960-kserve-79fc8564b5-gplxs"] Oct 30 16:43:27.804748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:27.804160 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:27.837221 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:27.836579 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ec445f-a902-4ed5-b917-a8460f8e4179" path="/var/lib/kubelet/pods/60ec445f-a902-4ed5-b917-a8460f8e4179/volumes" Oct 30 16:43:27.971761 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:27.970383 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.268535 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7"] Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.271711 4861 cpu_manager.go:414] "containerMap: removing container" podUID="51859589-1590-41e5-a1be-73680e4ced59" containerName="maas-api" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.271753 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51859589-1590-41e5-a1be-73680e4ced59" containerName="maas-api" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.271768 4861 cpu_manager.go:414] "containerMap: removing container" podUID="051f1a45-22fb-410a-93a8-d43cd5c730f1" containerName="maas-api" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.271776 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051f1a45-22fb-410a-93a8-d43cd5c730f1" containerName="maas-api" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.271792 4861 cpu_manager.go:414] "containerMap: removing container" podUID="a8f9abdc-9f33-4065-abac-a3fbf8ec8504" containerName="payload-pre-processing" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.271800 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f9abdc-9f33-4065-abac-a3fbf8ec8504" containerName="payload-pre-processing" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.271816 4861 cpu_manager.go:414] "containerMap: removing container" podUID="66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" containerName="istio-proxy" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.271823 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" containerName="istio-proxy" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.271868 4861 cpu_manager.go:414] "containerMap: removing container" podUID="60ec445f-a902-4ed5-b917-a8460f8e4179" containerName="main" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.271876 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ec445f-a902-4ed5-b917-a8460f8e4179" containerName="main" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.271896 4861 cpu_manager.go:414] "containerMap: removing container" podUID="5c265c02-98e3-4f69-b447-0811f146b548" containerName="maas-api" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.271904 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c265c02-98e3-4f69-b447-0811f146b548" containerName="maas-api" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.271921 4861 cpu_manager.go:414] "containerMap: removing container" podUID="50d1af4a-8f20-4200-85b7-615c9475d573" containerName="payload-processing" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.271927 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50d1af4a-8f20-4200-85b7-615c9475d573" containerName="payload-processing" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.271943 4861 cpu_manager.go:414] "containerMap: removing container" podUID="57333841-13b5-466b-8fc5-bf4c0b5ea649" containerName="payload-pre-processing" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.271949 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57333841-13b5-466b-8fc5-bf4c0b5ea649" containerName="payload-pre-processing" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.272100 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ec445f-a902-4ed5-b917-a8460f8e4179" containerName="main" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.272116 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c265c02-98e3-4f69-b447-0811f146b548" containerName="maas-api" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.272128 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="50d1af4a-8f20-4200-85b7-615c9475d573" containerName="payload-processing" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.272139 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="57333841-13b5-466b-8fc5-bf4c0b5ea649" containerName="payload-pre-processing" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.272151 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="51859589-1590-41e5-a1be-73680e4ced59" containerName="maas-api" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.272159 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="051f1a45-22fb-410a-93a8-d43cd5c730f1" containerName="maas-api" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.272166 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f9abdc-9f33-4065-abac-a3fbf8ec8504" containerName="payload-pre-processing" Oct 30 16:43:28.276602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.272174 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="66749430-c4d2-4fb7-a0a4-3c6ccd7cd2e7" containerName="istio-proxy" Oct 30 16:43:28.284837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.278962 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.301947 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.301254 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-843ab3-openshift-default-dockercfg-kz89h\"" Oct 30 16:43:28.319750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.315912 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-workload-certs\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.319750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.315957 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-workload-socket\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.319750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.316002 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-podinfo\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.319750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.316078 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m59ws\" (UniqueName: \"kubernetes.io/projected/8ab63f0b-0a42-4361-ad08-58464c7e65ab-kube-api-access-m59ws\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.319750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.316119 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istiod-ca-cert\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.319750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.316175 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-credential-socket\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.319750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.316282 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-data\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.319750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.316404 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-envoy\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.319750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.316451 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-token\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.326755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.323533 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7"] Oct 30 16:43:28.389476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.389403 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:28.417715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.417674 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-envoy\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.417939 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.417766 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-token\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.417939 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.417838 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-workload-certs\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.417939 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.417868 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-workload-socket\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.417939 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.417909 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-podinfo\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.418150 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.417985 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m59ws\" (UniqueName: \"kubernetes.io/projected/8ab63f0b-0a42-4361-ad08-58464c7e65ab-kube-api-access-m59ws\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.418150 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.418027 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istiod-ca-cert\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.418150 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.418086 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-credential-socket\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.418279 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.418200 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-data\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.459825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.418762 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-data\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.459825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.419469 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-workload-certs\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.459825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.420059 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-workload-socket\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.459825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.428676 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istiod-ca-cert\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.459825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.429307 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-credential-socket\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.479761 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.477358 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-podinfo\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.490650 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.490568 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-envoy\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.491906 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.491841 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m59ws\" (UniqueName: \"kubernetes.io/projected/8ab63f0b-0a42-4361-ad08-58464c7e65ab-kube-api-access-m59ws\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.518916 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.515412 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-token\") pod \"e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.610698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.610606 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:28.703503 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.701028 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:28.942053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.942028 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7"] Oct 30 16:43:28.942966 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:43:28.942938 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab63f0b_0a42_4361_ad08_58464c7e65ab.slice/crio-0f5d50a64e5be1d5b561ee484c3f659fc7d12e1bffb5b9348e203b8f07fc2c56 WatchSource:0}: Error finding container 0f5d50a64e5be1d5b561ee484c3f659fc7d12e1bffb5b9348e203b8f07fc2c56: Status 404 returned error can't find the container with id 0f5d50a64e5be1d5b561ee484c3f659fc7d12e1bffb5b9348e203b8f07fc2c56 Oct 30 16:43:28.945524 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.945494 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:43:28.945584 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.945560 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:43:28.945621 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:28.945589 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:43:29.585590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:29.585562 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:29.684456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:29.673994 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" event={"ID":"8ab63f0b-0a42-4361-ad08-58464c7e65ab","Type":"ContainerStarted","Data":"2a9faf03ba898ac4ce5e3f6d54207081d5c965ad22207174e7bb726cd70eeaef"} Oct 30 16:43:29.685110 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:29.685076 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" event={"ID":"8ab63f0b-0a42-4361-ad08-58464c7e65ab","Type":"ContainerStarted","Data":"0f5d50a64e5be1d5b561ee484c3f659fc7d12e1bffb5b9348e203b8f07fc2c56"} Oct 30 16:43:29.736217 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:29.736157 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" podStartSLOduration=1.736140926 podStartE2EDuration="1.736140926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:29.734078869 +0000 UTC m=+1240.076752936" watchObservedRunningTime="2026-10-30 16:43:29.736140926 +0000 UTC m=+1240.078814963" Oct 30 16:43:30.611113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:30.611083 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:30.617532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:30.617501 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:30.687463 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:30.687428 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:30.688461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:30.688437 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:43:31.470461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:31.470427 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:31.612468 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:31.612435 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:32.150019 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:32.149990 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:32.285090 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:32.285060 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:33.034734 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:33.034694 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:33.186960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:33.186931 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:33.342979 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:33.342907 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:33.548312 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:33.548279 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:33.944351 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:33.944321 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:34.593935 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:34.593896 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:34.979513 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:34.979482 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v"] Oct 30 16:43:34.985265 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:34.985242 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.010048 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.010020 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v"] Oct 30 16:43:35.018046 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.018010 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7c23a4-openshift-default-dockercfg-2mgnr\"" Oct 30 16:43:35.065780 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.065710 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:35.114400 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.114369 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-istio-envoy\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.114590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.114432 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-workload-certs\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.114590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.114457 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5026254-8aef-43a6-b079-072241e3b7b4-istio-token\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.114590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.114528 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-istio-data\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.114590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.114572 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-workload-socket\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.114874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.114619 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5026254-8aef-43a6-b079-072241e3b7b4-istio-podinfo\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.114874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.114638 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-credential-socket\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.114874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.114655 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2v4l\" (UniqueName: \"kubernetes.io/projected/e5026254-8aef-43a6-b079-072241e3b7b4-kube-api-access-j2v4l\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.114874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.114754 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5026254-8aef-43a6-b079-072241e3b7b4-istiod-ca-cert\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.210315 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.210287 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:35.216008 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.215983 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5026254-8aef-43a6-b079-072241e3b7b4-istiod-ca-cert\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.216155 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.216029 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-istio-envoy\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.216155 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.216078 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-workload-certs\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.216155 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.216101 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5026254-8aef-43a6-b079-072241e3b7b4-istio-token\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.216332 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.216186 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-istio-data\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.216332 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.216239 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-workload-socket\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.216332 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.216296 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5026254-8aef-43a6-b079-072241e3b7b4-istio-podinfo\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.216332 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.216326 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-credential-socket\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.216551 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.216348 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2v4l\" (UniqueName: \"kubernetes.io/projected/e5026254-8aef-43a6-b079-072241e3b7b4-kube-api-access-j2v4l\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.216610 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.216566 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-workload-certs\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.216673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.216639 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-istio-data\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.216914 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.216888 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-workload-socket\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.216914 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.216902 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5026254-8aef-43a6-b079-072241e3b7b4-istiod-ca-cert\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.217047 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.216960 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-credential-socket\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.218643 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.218623 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-istio-envoy\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.218995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.218977 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5026254-8aef-43a6-b079-072241e3b7b4-istio-podinfo\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.241595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.241517 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5026254-8aef-43a6-b079-072241e3b7b4-istio-token\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.241951 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.241923 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2v4l\" (UniqueName: \"kubernetes.io/projected/e5026254-8aef-43a6-b079-072241e3b7b4-kube-api-access-j2v4l\") pod \"e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.306912 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.306630 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:35.928886 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.928857 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v"] Oct 30 16:43:35.929508 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:43:35.929485 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5026254_8aef_43a6_b079_072241e3b7b4.slice/crio-1c41ab7df25c9154e65c6630ca30db6f60281e1e3771400296b588bcf0fdf22a WatchSource:0}: Error finding container 1c41ab7df25c9154e65c6630ca30db6f60281e1e3771400296b588bcf0fdf22a: Status 404 returned error can't find the container with id 1c41ab7df25c9154e65c6630ca30db6f60281e1e3771400296b588bcf0fdf22a Oct 30 16:43:35.932322 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.932289 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:43:35.932408 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.932353 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:43:35.932408 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:35.932382 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:43:36.102323 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:36.102288 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:36.832376 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:36.832345 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" event={"ID":"e5026254-8aef-43a6-b079-072241e3b7b4","Type":"ContainerStarted","Data":"0b2315a01b98298ca15952da55947fe27d96b433e97250ac1632c8889c65e916"} Oct 30 16:43:36.832540 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:36.832387 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" event={"ID":"e5026254-8aef-43a6-b079-072241e3b7b4","Type":"ContainerStarted","Data":"1c41ab7df25c9154e65c6630ca30db6f60281e1e3771400296b588bcf0fdf22a"} Oct 30 16:43:36.906072 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:36.906026 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" podStartSLOduration=2.9060126459999998 podStartE2EDuration="2.906012646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:36.89168037 +0000 UTC m=+1247.234354409" watchObservedRunningTime="2026-10-30 16:43:36.906012646 +0000 UTC m=+1247.248686761" Oct 30 16:43:37.307475 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:37.307442 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:37.313242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:37.313216 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:37.838215 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:37.838176 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:37.839028 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:37.839007 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:43:38.076436 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:38.076401 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:39.344360 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:39.344329 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:39.565461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:39.565429 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:39.993753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:39.989295 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:41.077123 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:41.077049 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:41.240692 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:41.240635 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:42.090489 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:42.090450 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:43.022608 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:43.022569 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:43.923121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:43.923090 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:44.070255 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:44.070221 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:44.223328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:44.223297 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:45.453970 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:45.453941 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw"] Oct 30 16:43:45.454414 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:45.454141 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" podUID="3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" containerName="istio-proxy" containerID="cri-o://71edbdf0be695dfe5025ca9e06b252e945507058e11cb8a0298952e860fef70d" gracePeriod=30 Oct 30 16:43:45.498336 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:45.498303 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb"] Oct 30 16:43:45.498545 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:45.498515 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" podUID="6abf00fa-1784-428a-9ddc-afa6963b2985" containerName="istio-proxy" containerID="cri-o://46784ebd12399423ca46aa4f2bfbdc0ab34564f66c3e12ae92026511ac88196f" gracePeriod=30 Oct 30 16:43:45.826412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:45.826384 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w"] Oct 30 16:43:45.830680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:45.830664 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:43:45.849065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:45.849038 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-843ab3-serving-cert\"" Oct 30 16:43:45.853586 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:45.853552 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w"] Oct 30 16:43:45.900683 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:45.900651 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sq5l\" (UniqueName: \"kubernetes.io/projected/e88f463a-7f54-49ba-94da-f6522fe52aa2-kube-api-access-5sq5l\") pod \"maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w\" (UID: \"e88f463a-7f54-49ba-94da-f6522fe52aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:43:45.900918 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:45.900842 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e88f463a-7f54-49ba-94da-f6522fe52aa2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w\" (UID: \"e88f463a-7f54-49ba-94da-f6522fe52aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:43:45.901083 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:45.901061 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e88f463a-7f54-49ba-94da-f6522fe52aa2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w\" (UID: \"e88f463a-7f54-49ba-94da-f6522fe52aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:43:45.901368 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:45.901349 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e88f463a-7f54-49ba-94da-f6522fe52aa2-metrics-certs\") pod \"maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w\" (UID: \"e88f463a-7f54-49ba-94da-f6522fe52aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:43:45.987955 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:45.987924 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:46.003036 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:46.003011 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e88f463a-7f54-49ba-94da-f6522fe52aa2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w\" (UID: \"e88f463a-7f54-49ba-94da-f6522fe52aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:43:46.003166 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:46.003152 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e88f463a-7f54-49ba-94da-f6522fe52aa2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w\" (UID: \"e88f463a-7f54-49ba-94da-f6522fe52aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:43:46.003234 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:46.003212 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e88f463a-7f54-49ba-94da-f6522fe52aa2-metrics-certs\") pod \"maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w\" (UID: \"e88f463a-7f54-49ba-94da-f6522fe52aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:43:46.003299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:46.003285 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sq5l\" (UniqueName: \"kubernetes.io/projected/e88f463a-7f54-49ba-94da-f6522fe52aa2-kube-api-access-5sq5l\") pod \"maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w\" (UID: \"e88f463a-7f54-49ba-94da-f6522fe52aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:43:46.003775 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:46.003746 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e88f463a-7f54-49ba-94da-f6522fe52aa2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w\" (UID: \"e88f463a-7f54-49ba-94da-f6522fe52aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:43:46.006087 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:46.006067 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e88f463a-7f54-49ba-94da-f6522fe52aa2-metrics-certs\") pod \"maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w\" (UID: \"e88f463a-7f54-49ba-94da-f6522fe52aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:43:46.006664 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:46.006646 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e88f463a-7f54-49ba-94da-f6522fe52aa2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w\" (UID: \"e88f463a-7f54-49ba-94da-f6522fe52aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:43:46.027622 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:46.027603 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sq5l\" (UniqueName: \"kubernetes.io/projected/e88f463a-7f54-49ba-94da-f6522fe52aa2-kube-api-access-5sq5l\") pod \"maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w\" (UID: \"e88f463a-7f54-49ba-94da-f6522fe52aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:43:46.142331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:46.142258 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:46.144609 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:46.144591 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:43:46.599428 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:46.599399 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w"] Oct 30 16:43:46.600382 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:43:46.600357 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88f463a_7f54_49ba_94da_f6522fe52aa2.slice/crio-5036c7dffd9b5b549b575be31cb621aa01897feb66fae604911045bc292e417a WatchSource:0}: Error finding container 5036c7dffd9b5b549b575be31cb621aa01897feb66fae604911045bc292e417a: Status 404 returned error can't find the container with id 5036c7dffd9b5b549b575be31cb621aa01897feb66fae604911045bc292e417a Oct 30 16:43:46.900979 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:46.900897 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" event={"ID":"e88f463a-7f54-49ba-94da-f6522fe52aa2","Type":"ContainerStarted","Data":"5036c7dffd9b5b549b575be31cb621aa01897feb66fae604911045bc292e417a"} Oct 30 16:43:47.908520 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:47.908481 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" event={"ID":"e88f463a-7f54-49ba-94da-f6522fe52aa2","Type":"ContainerStarted","Data":"b3be194a85186ff822573c74c43c39399c1d8906f29dc7b5bd42525a084548b6"} Oct 30 16:43:47.908986 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:47.908557 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:43:47.955358 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:47.955309 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" podStartSLOduration=2.5052889560000002 podStartE2EDuration="2.955295666s" totalImagesPullingTime="450.00671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:47.952094684 +0000 UTC m=+1258.294768726" watchObservedRunningTime="2026-10-30 16:43:47.955295666 +0000 UTC m=+1258.297969703" Oct 30 16:43:47.994755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:47.992697 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:49.296208 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:49.296175 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:49.990183 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:49.990153 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:50.173770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.173739 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:50.290627 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:43:50.290598 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:43:50.796076 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.796053 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:43:50.829887 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.829865 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:43:50.930551 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.930520 4861 generic.go:356] "Generic (PLEG): container finished" podID="6abf00fa-1784-428a-9ddc-afa6963b2985" containerID="46784ebd12399423ca46aa4f2bfbdc0ab34564f66c3e12ae92026511ac88196f" exitCode=0 Oct 30 16:43:50.930747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.930690 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" Oct 30 16:43:50.931051 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.931028 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" event={"ID":"6abf00fa-1784-428a-9ddc-afa6963b2985","Type":"ContainerDied","Data":"46784ebd12399423ca46aa4f2bfbdc0ab34564f66c3e12ae92026511ac88196f"} Oct 30 16:43:50.931150 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.931076 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb" event={"ID":"6abf00fa-1784-428a-9ddc-afa6963b2985","Type":"ContainerDied","Data":"55b398b7fe7a82f22f25b7b82058b985d77272cb5f3eb7887e8448e9ba09c662"} Oct 30 16:43:50.931150 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.931108 4861 scope.go:122] "RemoveContainer" containerID="46784ebd12399423ca46aa4f2bfbdc0ab34564f66c3e12ae92026511ac88196f" Oct 30 16:43:50.933147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.933121 4861 generic.go:356] "Generic (PLEG): container finished" podID="3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" containerID="71edbdf0be695dfe5025ca9e06b252e945507058e11cb8a0298952e860fef70d" exitCode=0 Oct 30 16:43:50.933348 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.933329 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" event={"ID":"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec","Type":"ContainerDied","Data":"71edbdf0be695dfe5025ca9e06b252e945507058e11cb8a0298952e860fef70d"} Oct 30 16:43:50.933397 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.933381 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" event={"ID":"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec","Type":"ContainerDied","Data":"78fb9b6e919848e227dafc4911ede46b1b5c4c91944b433268e31204e7ce5034"} Oct 30 16:43:50.933474 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.933461 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw" Oct 30 16:43:50.941937 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.941918 4861 scope.go:122] "RemoveContainer" containerID="46784ebd12399423ca46aa4f2bfbdc0ab34564f66c3e12ae92026511ac88196f" Oct 30 16:43:50.942211 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:43:50.942193 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46784ebd12399423ca46aa4f2bfbdc0ab34564f66c3e12ae92026511ac88196f\": container with ID starting with 46784ebd12399423ca46aa4f2bfbdc0ab34564f66c3e12ae92026511ac88196f not found: ID does not exist" containerID="46784ebd12399423ca46aa4f2bfbdc0ab34564f66c3e12ae92026511ac88196f" Oct 30 16:43:50.942288 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.942221 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46784ebd12399423ca46aa4f2bfbdc0ab34564f66c3e12ae92026511ac88196f"} err="failed to get container status \"46784ebd12399423ca46aa4f2bfbdc0ab34564f66c3e12ae92026511ac88196f\": rpc error: code = NotFound desc = could not find container \"46784ebd12399423ca46aa4f2bfbdc0ab34564f66c3e12ae92026511ac88196f\": container with ID starting with 46784ebd12399423ca46aa4f2bfbdc0ab34564f66c3e12ae92026511ac88196f not found: ID does not exist" Oct 30 16:43:50.942288 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.942243 4861 scope.go:122] "RemoveContainer" containerID="71edbdf0be695dfe5025ca9e06b252e945507058e11cb8a0298952e860fef70d" Oct 30 16:43:50.952794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.952778 4861 scope.go:122] "RemoveContainer" containerID="71edbdf0be695dfe5025ca9e06b252e945507058e11cb8a0298952e860fef70d" Oct 30 16:43:50.953081 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:43:50.953062 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71edbdf0be695dfe5025ca9e06b252e945507058e11cb8a0298952e860fef70d\": container with ID starting with 71edbdf0be695dfe5025ca9e06b252e945507058e11cb8a0298952e860fef70d not found: ID does not exist" containerID="71edbdf0be695dfe5025ca9e06b252e945507058e11cb8a0298952e860fef70d" Oct 30 16:43:50.953152 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.953089 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71edbdf0be695dfe5025ca9e06b252e945507058e11cb8a0298952e860fef70d"} err="failed to get container status \"71edbdf0be695dfe5025ca9e06b252e945507058e11cb8a0298952e860fef70d\": rpc error: code = NotFound desc = could not find container \"71edbdf0be695dfe5025ca9e06b252e945507058e11cb8a0298952e860fef70d\": container with ID starting with 71edbdf0be695dfe5025ca9e06b252e945507058e11cb8a0298952e860fef70d not found: ID does not exist" Oct 30 16:43:50.985451 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.985418 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-podinfo\") pod \"6abf00fa-1784-428a-9ddc-afa6963b2985\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " Oct 30 16:43:50.985560 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.985521 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abf00fa-1784-428a-9ddc-afa6963b2985-kube-api-access-8dwzg\" (UniqueName: \"kubernetes.io/projected/6abf00fa-1784-428a-9ddc-afa6963b2985-kube-api-access-8dwzg\") pod \"6abf00fa-1784-428a-9ddc-afa6963b2985\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " Oct 30 16:43:50.985623 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.985610 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-token\" (UniqueName: \"kubernetes.io/projected/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-token\") pod \"6abf00fa-1784-428a-9ddc-afa6963b2985\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " Oct 30 16:43:50.985675 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.985649 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-workload-certs\") pod \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " Oct 30 16:43:50.985740 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.985687 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-envoy\") pod \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " Oct 30 16:43:50.985797 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.985758 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-data\") pod \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " Oct 30 16:43:50.985852 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.985826 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-envoy\") pod \"6abf00fa-1784-428a-9ddc-afa6963b2985\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " Oct 30 16:43:50.985905 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.985892 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-data\") pod \"6abf00fa-1784-428a-9ddc-afa6963b2985\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " Oct 30 16:43:50.985957 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.985933 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6abf00fa-1784-428a-9ddc-afa6963b2985-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6abf00fa-1784-428a-9ddc-afa6963b2985-istiod-ca-cert\") pod \"6abf00fa-1784-428a-9ddc-afa6963b2985\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " Oct 30 16:43:50.986009 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.985973 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istiod-ca-cert\") pod \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " Oct 30 16:43:50.986063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.986011 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-podinfo\") pod \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " Oct 30 16:43:50.986063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.986022 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-workload-certs" pod "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" (UID: "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:50.986063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.986041 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-token\" (UniqueName: \"kubernetes.io/projected/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-token\") pod \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " Oct 30 16:43:50.986207 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.986115 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-credential-socket\") pod \"6abf00fa-1784-428a-9ddc-afa6963b2985\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " Oct 30 16:43:50.986207 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.986163 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-workload-certs\") pod \"6abf00fa-1784-428a-9ddc-afa6963b2985\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " Oct 30 16:43:50.986307 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.986204 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-kube-api-access-cbsvj\" (UniqueName: \"kubernetes.io/projected/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-kube-api-access-cbsvj\") pod \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " Oct 30 16:43:50.986307 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.986235 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-workload-socket\") pod \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " Oct 30 16:43:50.986307 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.986275 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-workload-socket\") pod \"6abf00fa-1784-428a-9ddc-afa6963b2985\" (UID: \"6abf00fa-1784-428a-9ddc-afa6963b2985\") " Oct 30 16:43:50.986453 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.986369 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6abf00fa-1784-428a-9ddc-afa6963b2985-istiod-ca-cert" pod "6abf00fa-1784-428a-9ddc-afa6963b2985" (UID: "6abf00fa-1784-428a-9ddc-afa6963b2985"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:50.986453 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.986369 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-data" pod "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" (UID: "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:50.986453 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.986386 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-credential-socket\") pod \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\" (UID: \"3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec\") " Oct 30 16:43:50.987627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.986991 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-workload-certs" pod "6abf00fa-1784-428a-9ddc-afa6963b2985" (UID: "6abf00fa-1784-428a-9ddc-afa6963b2985"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:50.987627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.987205 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-credential-socket" pod "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" (UID: "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:50.987627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.987365 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-workload-socket" pod "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" (UID: "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:50.987627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.987542 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-workload-socket" pod "6abf00fa-1784-428a-9ddc-afa6963b2985" (UID: "6abf00fa-1784-428a-9ddc-afa6963b2985"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:50.987922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.987896 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-data" pod "6abf00fa-1784-428a-9ddc-afa6963b2985" (UID: "6abf00fa-1784-428a-9ddc-afa6963b2985"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:50.987990 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.987959 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-credential-socket" pod "6abf00fa-1784-428a-9ddc-afa6963b2985" (UID: "6abf00fa-1784-428a-9ddc-afa6963b2985"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:50.987990 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.987958 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istiod-ca-cert" pod "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" (UID: "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:50.988151 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.988109 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-podinfo" pod "6abf00fa-1784-428a-9ddc-afa6963b2985" (UID: "6abf00fa-1784-428a-9ddc-afa6963b2985"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:43:50.988366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.988317 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:50.988366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.988346 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:50.988366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.988359 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:50.988366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.988370 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6abf00fa-1784-428a-9ddc-afa6963b2985-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:50.988366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.988382 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:50.988366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.988394 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:50.988366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.988407 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:50.988366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.988420 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:50.988366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.988434 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:50.988981 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.988447 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:50.988981 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.988459 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:50.988981 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.988622 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abf00fa-1784-428a-9ddc-afa6963b2985-kube-api-access-8dwzg" pod "6abf00fa-1784-428a-9ddc-afa6963b2985" (UID: "6abf00fa-1784-428a-9ddc-afa6963b2985"). InnerVolumeSpecName "kube-api-access-8dwzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:50.988981 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.988648 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-token" pod "6abf00fa-1784-428a-9ddc-afa6963b2985" (UID: "6abf00fa-1784-428a-9ddc-afa6963b2985"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:50.990028 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.990001 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-podinfo" pod "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" (UID: "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:43:50.990144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.990077 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-envoy" pod "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" (UID: "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:50.990144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.990085 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-token" pod "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" (UID: "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:50.990144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.990113 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-envoy" pod "6abf00fa-1784-428a-9ddc-afa6963b2985" (UID: "6abf00fa-1784-428a-9ddc-afa6963b2985"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:50.990767 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:50.990751 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-kube-api-access-cbsvj" pod "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" (UID: "3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec"). InnerVolumeSpecName "kube-api-access-cbsvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:51.090412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:51.090333 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:51.090412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:51.090360 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:51.090412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:51.090368 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6abf00fa-1784-428a-9ddc-afa6963b2985-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:51.090412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:51.090377 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:51.090412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:51.090385 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:51.090412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:51.090394 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbsvj\" (UniqueName: \"kubernetes.io/projected/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec-kube-api-access-cbsvj\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:51.090412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:51.090407 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dwzg\" (UniqueName: \"kubernetes.io/projected/6abf00fa-1784-428a-9ddc-afa6963b2985-kube-api-access-8dwzg\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:43:51.280329 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:51.280304 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb"] Oct 30 16:43:51.293202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:51.293179 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-pwxkb"] Oct 30 16:43:51.328372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:51.328341 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw"] Oct 30 16:43:51.333201 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:51.333180 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-2gfgw"] Oct 30 16:43:51.808180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:51.808106 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" path="/var/lib/kubelet/pods/3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec/volumes" Oct 30 16:43:51.808574 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:51.808559 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6abf00fa-1784-428a-9ddc-afa6963b2985" path="/var/lib/kubelet/pods/6abf00fa-1784-428a-9ddc-afa6963b2985/volumes" Oct 30 16:43:51.908113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:51.908086 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:52.682182 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:52.682154 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:52.846181 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:52.846152 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:53.545341 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.545312 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q"] Oct 30 16:43:53.545953 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.545939 4861 cpu_manager.go:414] "containerMap: removing container" podUID="6abf00fa-1784-428a-9ddc-afa6963b2985" containerName="istio-proxy" Oct 30 16:43:53.546026 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.545954 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6abf00fa-1784-428a-9ddc-afa6963b2985" containerName="istio-proxy" Oct 30 16:43:53.546026 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.545972 4861 cpu_manager.go:414] "containerMap: removing container" podUID="3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" containerName="istio-proxy" Oct 30 16:43:53.546026 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.545978 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" containerName="istio-proxy" Oct 30 16:43:53.546137 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.546063 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="6abf00fa-1784-428a-9ddc-afa6963b2985" containerName="istio-proxy" Oct 30 16:43:53.546137 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.546071 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f90cfb5-ffa8-40ca-af8c-64c24bc3e3ec" containerName="istio-proxy" Oct 30 16:43:53.551152 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.551134 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:43:53.584069 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.584042 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7c23a4-serving-cert\"" Oct 30 16:43:53.589964 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.589932 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q"] Oct 30 16:43:53.631411 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.631381 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8112653e-e8ad-4214-add5-acf873438708-metrics-certs\") pod \"maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q\" (UID: \"8112653e-e8ad-4214-add5-acf873438708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:43:53.631560 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.631489 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmhjn\" (UniqueName: \"kubernetes.io/projected/8112653e-e8ad-4214-add5-acf873438708-kube-api-access-nmhjn\") pod \"maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q\" (UID: \"8112653e-e8ad-4214-add5-acf873438708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:43:53.631618 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.631568 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8112653e-e8ad-4214-add5-acf873438708-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q\" (UID: \"8112653e-e8ad-4214-add5-acf873438708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:43:53.631693 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.631681 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8112653e-e8ad-4214-add5-acf873438708-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q\" (UID: \"8112653e-e8ad-4214-add5-acf873438708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:43:53.717443 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.717417 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:53.732917 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.732894 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8112653e-e8ad-4214-add5-acf873438708-metrics-certs\") pod \"maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q\" (UID: \"8112653e-e8ad-4214-add5-acf873438708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:43:53.733039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.732952 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmhjn\" (UniqueName: \"kubernetes.io/projected/8112653e-e8ad-4214-add5-acf873438708-kube-api-access-nmhjn\") pod \"maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q\" (UID: \"8112653e-e8ad-4214-add5-acf873438708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:43:53.733086 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.733038 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8112653e-e8ad-4214-add5-acf873438708-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q\" (UID: \"8112653e-e8ad-4214-add5-acf873438708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:43:53.733141 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.733119 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8112653e-e8ad-4214-add5-acf873438708-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q\" (UID: \"8112653e-e8ad-4214-add5-acf873438708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:43:53.733639 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.733618 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8112653e-e8ad-4214-add5-acf873438708-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q\" (UID: \"8112653e-e8ad-4214-add5-acf873438708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:43:53.735509 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.735477 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8112653e-e8ad-4214-add5-acf873438708-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q\" (UID: \"8112653e-e8ad-4214-add5-acf873438708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:43:53.735584 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.735564 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8112653e-e8ad-4214-add5-acf873438708-metrics-certs\") pod \"maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q\" (UID: \"8112653e-e8ad-4214-add5-acf873438708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:43:53.757142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.757119 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmhjn\" (UniqueName: \"kubernetes.io/projected/8112653e-e8ad-4214-add5-acf873438708-kube-api-access-nmhjn\") pod \"maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q\" (UID: \"8112653e-e8ad-4214-add5-acf873438708\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:43:53.868209 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.868113 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:43:53.905797 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.905682 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:53.932773 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:53.931435 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:43:54.250262 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:54.250234 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q"] Oct 30 16:43:54.251123 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:43:54.251099 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8112653e_e8ad_4214_add5_acf873438708.slice/crio-b55a59d173ecc5ea45691266ee34437acaec14c4749357364b8119a3a5874d06 WatchSource:0}: Error finding container b55a59d173ecc5ea45691266ee34437acaec14c4749357364b8119a3a5874d06: Status 404 returned error can't find the container with id b55a59d173ecc5ea45691266ee34437acaec14c4749357364b8119a3a5874d06 Oct 30 16:43:55.036411 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:55.036356 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" event={"ID":"8112653e-e8ad-4214-add5-acf873438708","Type":"ContainerStarted","Data":"a35fc09da9a0feb39458a3db8be1780966592052f35094c2249d52fb6af08547"} Oct 30 16:43:55.036940 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:55.036456 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" event={"ID":"8112653e-e8ad-4214-add5-acf873438708","Type":"ContainerStarted","Data":"b55a59d173ecc5ea45691266ee34437acaec14c4749357364b8119a3a5874d06"} Oct 30 16:43:55.036940 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:55.036607 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:43:55.097263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:55.082185 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" podStartSLOduration=1.7287383649999999 podStartE2EDuration="2.082165936s" totalImagesPullingTime="353.427571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:55.081396283 +0000 UTC m=+1265.424070325" watchObservedRunningTime="2026-10-30 16:43:55.082165936 +0000 UTC m=+1265.424839976" Oct 30 16:43:55.873750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:55.867743 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:56.863697 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:56.863659 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:57.139014 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.138939 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc"] Oct 30 16:43:57.144103 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.144075 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.160350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.160323 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-504442-openshift-default-dockercfg-pvqxp\"" Oct 30 16:43:57.177442 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.177416 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc"] Oct 30 16:43:57.231127 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.231066 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:57.234590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.234566 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-podinfo\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.234706 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.234642 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-workload-socket\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.234781 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.234760 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqscm\" (UniqueName: \"kubernetes.io/projected/e3c4a663-826a-4d68-9208-e16cb76839b5-kube-api-access-bqscm\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.234822 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.234807 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-credential-socket\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.234878 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.234863 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-envoy\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.234917 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.234887 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-data\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.234917 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.234911 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c4a663-826a-4d68-9208-e16cb76839b5-istiod-ca-cert\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.235011 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.234991 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-workload-certs\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.235138 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.235124 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-token\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.342746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.341798 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-token\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.342746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.341909 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-podinfo\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.342746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.342015 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-workload-socket\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.342746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.342079 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqscm\" (UniqueName: \"kubernetes.io/projected/e3c4a663-826a-4d68-9208-e16cb76839b5-kube-api-access-bqscm\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.342746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.342124 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-credential-socket\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.342746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.342156 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-envoy\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.342746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.342181 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-data\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.342746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.342219 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c4a663-826a-4d68-9208-e16cb76839b5-istiod-ca-cert\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.342746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.342276 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-workload-certs\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.343377 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.342889 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-workload-certs\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.361749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.344076 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-credential-socket\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.361749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.344340 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-workload-socket\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.361749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.344855 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-data\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.361749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.351499 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c4a663-826a-4d68-9208-e16cb76839b5-istiod-ca-cert\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.361749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.360644 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-podinfo\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.411799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.391463 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-envoy\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.411799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.395244 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-token\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.411799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.402377 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqscm\" (UniqueName: \"kubernetes.io/projected/e3c4a663-826a-4d68-9208-e16cb76839b5-kube-api-access-bqscm\") pod \"e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.440751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.440012 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:57.476745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.471307 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:57.788558 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.788528 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:57.955781 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.955694 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:57.985371 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.985340 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc"] Oct 30 16:43:57.985718 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:43:57.985690 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c4a663_826a_4d68_9208_e16cb76839b5.slice/crio-48d6c575f98bc8e70c1372fde46e465a7f738b894631c478522c347c0ff361da WatchSource:0}: Error finding container 48d6c575f98bc8e70c1372fde46e465a7f738b894631c478522c347c0ff361da: Status 404 returned error can't find the container with id 48d6c575f98bc8e70c1372fde46e465a7f738b894631c478522c347c0ff361da Oct 30 16:43:57.988558 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.988531 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:43:57.988620 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.988593 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:43:57.988672 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:57.988623 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:43:58.160436 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:58.160398 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg"] Oct 30 16:43:58.160707 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:58.160665 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" podUID="38ac300a-fa56-4c84-8c54-ff4d677e9555" containerName="istio-proxy" containerID="cri-o://5961c9cfb09f31cacd458122e3cacecd72bdb5c0b3e5ca2f314eaa6acc15dbaa" gracePeriod=30 Oct 30 16:43:58.186806 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:58.185566 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:58.194356 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:58.194320 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" event={"ID":"e3c4a663-826a-4d68-9208-e16cb76839b5","Type":"ContainerStarted","Data":"48d6c575f98bc8e70c1372fde46e465a7f738b894631c478522c347c0ff361da"} Oct 30 16:43:58.389062 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:58.388988 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:58.976406 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:58.976379 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:59.204503 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:59.204465 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" event={"ID":"e3c4a663-826a-4d68-9208-e16cb76839b5","Type":"ContainerStarted","Data":"6fbed1c15fbce1d4d93b5dde9ac59029a7b9277a09599863fa88ac2558bb4886"} Oct 30 16:43:59.268250 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:59.268157 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" podStartSLOduration=2.268144507 podStartE2EDuration="2.268144507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:59.263770983 +0000 UTC m=+1269.606445010" watchObservedRunningTime="2026-10-30 16:43:59.268144507 +0000 UTC m=+1269.610818545" Oct 30 16:43:59.451423 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:59.451389 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:43:59.520364 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:59.520273 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:59.691747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:59.691261 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:43:59.876648 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:43:59.876561 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:00.213388 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:00.213303 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:44:00.214448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:00.214418 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:44:00.248190 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:00.248166 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:01.057182 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:01.057153 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:44:01.803244 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:01.803214 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:01.948786 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:01.948754 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:02.211653 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:02.211565 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:02.723219 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:02.723182 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:02.879307 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:02.879244 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:03.441338 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.441258 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:44:03.481337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.481272 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-token\" (UniqueName: \"kubernetes.io/projected/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-token\") pod \"38ac300a-fa56-4c84-8c54-ff4d677e9555\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " Oct 30 16:44:03.481337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.481326 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ac300a-fa56-4c84-8c54-ff4d677e9555-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ac300a-fa56-4c84-8c54-ff4d677e9555-istiod-ca-cert\") pod \"38ac300a-fa56-4c84-8c54-ff4d677e9555\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " Oct 30 16:44:03.481716 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.481443 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-workload-certs\") pod \"38ac300a-fa56-4c84-8c54-ff4d677e9555\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " Oct 30 16:44:03.481716 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.481510 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-envoy\") pod \"38ac300a-fa56-4c84-8c54-ff4d677e9555\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " Oct 30 16:44:03.482537 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.482173 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-workload-certs" pod "38ac300a-fa56-4c84-8c54-ff4d677e9555" (UID: "38ac300a-fa56-4c84-8c54-ff4d677e9555"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:03.482689 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.482337 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ac300a-fa56-4c84-8c54-ff4d677e9555-istiod-ca-cert" pod "38ac300a-fa56-4c84-8c54-ff4d677e9555" (UID: "38ac300a-fa56-4c84-8c54-ff4d677e9555"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:03.482689 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.482259 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ac300a-fa56-4c84-8c54-ff4d677e9555-kube-api-access-ngkg2\" (UniqueName: \"kubernetes.io/projected/38ac300a-fa56-4c84-8c54-ff4d677e9555-kube-api-access-ngkg2\") pod \"38ac300a-fa56-4c84-8c54-ff4d677e9555\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " Oct 30 16:44:03.482689 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.482614 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-podinfo\") pod \"38ac300a-fa56-4c84-8c54-ff4d677e9555\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " Oct 30 16:44:03.482689 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.482677 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-credential-socket\") pod \"38ac300a-fa56-4c84-8c54-ff4d677e9555\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " Oct 30 16:44:03.483083 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.482759 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-workload-socket\") pod \"38ac300a-fa56-4c84-8c54-ff4d677e9555\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " Oct 30 16:44:03.483196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.483179 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-data\") pod \"38ac300a-fa56-4c84-8c54-ff4d677e9555\" (UID: \"38ac300a-fa56-4c84-8c54-ff4d677e9555\") " Oct 30 16:44:03.483552 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.483523 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-data" pod "38ac300a-fa56-4c84-8c54-ff4d677e9555" (UID: "38ac300a-fa56-4c84-8c54-ff4d677e9555"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:03.484818 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.483942 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:03.484818 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.483969 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ac300a-fa56-4c84-8c54-ff4d677e9555-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:03.484818 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.483980 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:03.484818 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.484790 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-credential-socket" pod "38ac300a-fa56-4c84-8c54-ff4d677e9555" (UID: "38ac300a-fa56-4c84-8c54-ff4d677e9555"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:03.485124 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.485104 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-workload-socket" pod "38ac300a-fa56-4c84-8c54-ff4d677e9555" (UID: "38ac300a-fa56-4c84-8c54-ff4d677e9555"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:03.485737 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.485702 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-podinfo" pod "38ac300a-fa56-4c84-8c54-ff4d677e9555" (UID: "38ac300a-fa56-4c84-8c54-ff4d677e9555"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:44:03.485837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.485705 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-token" pod "38ac300a-fa56-4c84-8c54-ff4d677e9555" (UID: "38ac300a-fa56-4c84-8c54-ff4d677e9555"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:03.485907 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.485889 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ac300a-fa56-4c84-8c54-ff4d677e9555-kube-api-access-ngkg2" pod "38ac300a-fa56-4c84-8c54-ff4d677e9555" (UID: "38ac300a-fa56-4c84-8c54-ff4d677e9555"). InnerVolumeSpecName "kube-api-access-ngkg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:03.487279 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.487116 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-envoy" pod "38ac300a-fa56-4c84-8c54-ff4d677e9555" (UID: "38ac300a-fa56-4c84-8c54-ff4d677e9555"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:03.515282 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.515247 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:03.585285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.585254 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:03.585447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.585305 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngkg2\" (UniqueName: \"kubernetes.io/projected/38ac300a-fa56-4c84-8c54-ff4d677e9555-kube-api-access-ngkg2\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:03.585447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.585317 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:03.585447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.585327 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:03.585447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.585337 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ac300a-fa56-4c84-8c54-ff4d677e9555-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:03.585447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.585346 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ac300a-fa56-4c84-8c54-ff4d677e9555-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:03.657533 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.657504 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b"] Oct 30 16:44:03.658157 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.658143 4861 cpu_manager.go:414] "containerMap: removing container" podUID="38ac300a-fa56-4c84-8c54-ff4d677e9555" containerName="istio-proxy" Oct 30 16:44:03.658208 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.658159 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ac300a-fa56-4c84-8c54-ff4d677e9555" containerName="istio-proxy" Oct 30 16:44:03.658241 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.658208 4861 cpu_manager.go:414] "containerMap: removing container" podUID="38ac300a-fa56-4c84-8c54-ff4d677e9555" containerName="istio-proxy" Oct 30 16:44:03.658241 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.658214 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ac300a-fa56-4c84-8c54-ff4d677e9555" containerName="istio-proxy" Oct 30 16:44:03.658325 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.658316 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ac300a-fa56-4c84-8c54-ff4d677e9555" containerName="istio-proxy" Oct 30 16:44:03.662123 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.662106 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" Oct 30 16:44:03.670272 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.670247 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:03.678340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.678304 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-504442\"" Oct 30 16:44:03.678340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.678333 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-504442-dockercfg-zrjqx\"" Oct 30 16:44:03.682454 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.682434 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b"] Oct 30 16:44:03.807751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.799014 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fecce1f-7675-40f7-9e2f-a849ceb4bc04-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b\" (UID: \"2fecce1f-7675-40f7-9e2f-a849ceb4bc04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" Oct 30 16:44:03.849039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.848981 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56dzn\" (UniqueName: \"kubernetes.io/projected/2fecce1f-7675-40f7-9e2f-a849ceb4bc04-kube-api-access-56dzn\") pod \"payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b\" (UID: \"2fecce1f-7675-40f7-9e2f-a849ceb4bc04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" Oct 30 16:44:03.896862 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.895942 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c"] Oct 30 16:44:03.898527 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.897819 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ac300a-fa56-4c84-8c54-ff4d677e9555" containerName="istio-proxy" Oct 30 16:44:03.902806 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.902183 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c"] Oct 30 16:44:03.902806 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.902291 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" Oct 30 16:44:03.953998 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.953956 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fecce1f-7675-40f7-9e2f-a849ceb4bc04-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b\" (UID: \"2fecce1f-7675-40f7-9e2f-a849ceb4bc04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" Oct 30 16:44:03.959783 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.954244 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56dzn\" (UniqueName: \"kubernetes.io/projected/2fecce1f-7675-40f7-9e2f-a849ceb4bc04-kube-api-access-56dzn\") pod \"payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b\" (UID: \"2fecce1f-7675-40f7-9e2f-a849ceb4bc04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" Oct 30 16:44:03.959783 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:03.955471 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fecce1f-7675-40f7-9e2f-a849ceb4bc04-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b\" (UID: \"2fecce1f-7675-40f7-9e2f-a849ceb4bc04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" Oct 30 16:44:04.038113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.037161 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56dzn\" (UniqueName: \"kubernetes.io/projected/2fecce1f-7675-40f7-9e2f-a849ceb4bc04-kube-api-access-56dzn\") pod \"payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b\" (UID: \"2fecce1f-7675-40f7-9e2f-a849ceb4bc04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" Oct 30 16:44:04.056214 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.056103 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frptg\" (UniqueName: \"kubernetes.io/projected/ed177420-80fa-483f-a97a-62286e7f20fd-kube-api-access-frptg\") pod \"payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c\" (UID: \"ed177420-80fa-483f-a97a-62286e7f20fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" Oct 30 16:44:04.057707 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.057507 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed177420-80fa-483f-a97a-62286e7f20fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c\" (UID: \"ed177420-80fa-483f-a97a-62286e7f20fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" Oct 30 16:44:04.159358 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.159323 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed177420-80fa-483f-a97a-62286e7f20fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c\" (UID: \"ed177420-80fa-483f-a97a-62286e7f20fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" Oct 30 16:44:04.159532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.159520 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frptg\" (UniqueName: \"kubernetes.io/projected/ed177420-80fa-483f-a97a-62286e7f20fd-kube-api-access-frptg\") pod \"payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c\" (UID: \"ed177420-80fa-483f-a97a-62286e7f20fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" Oct 30 16:44:04.160017 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.159999 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed177420-80fa-483f-a97a-62286e7f20fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c\" (UID: \"ed177420-80fa-483f-a97a-62286e7f20fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" Oct 30 16:44:04.183401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.183371 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frptg\" (UniqueName: \"kubernetes.io/projected/ed177420-80fa-483f-a97a-62286e7f20fd-kube-api-access-frptg\") pod \"payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c\" (UID: \"ed177420-80fa-483f-a97a-62286e7f20fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" Oct 30 16:44:04.276419 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.276391 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" Oct 30 16:44:04.277855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.277821 4861 generic.go:356] "Generic (PLEG): container finished" podID="38ac300a-fa56-4c84-8c54-ff4d677e9555" containerID="5961c9cfb09f31cacd458122e3cacecd72bdb5c0b3e5ca2f314eaa6acc15dbaa" exitCode=0 Oct 30 16:44:04.277982 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.277863 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" event={"ID":"38ac300a-fa56-4c84-8c54-ff4d677e9555","Type":"ContainerDied","Data":"5961c9cfb09f31cacd458122e3cacecd72bdb5c0b3e5ca2f314eaa6acc15dbaa"} Oct 30 16:44:04.277982 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.277937 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" event={"ID":"38ac300a-fa56-4c84-8c54-ff4d677e9555","Type":"ContainerDied","Data":"aa9cfc520ce88f534834b68f60083b5eb4a7a7a77bd1ed2d2b120db146cb3697"} Oct 30 16:44:04.277982 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.277965 4861 scope.go:122] "RemoveContainer" containerID="5961c9cfb09f31cacd458122e3cacecd72bdb5c0b3e5ca2f314eaa6acc15dbaa" Oct 30 16:44:04.278146 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.278137 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg" Oct 30 16:44:04.391813 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.381414 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" Oct 30 16:44:04.497991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.497952 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg"] Oct 30 16:44:04.498158 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.498014 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-82381960-openshift-default-68d7558b65-5pgqg"] Oct 30 16:44:04.506872 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.506240 4861 scope.go:122] "RemoveContainer" containerID="5edecec21ad47c959bc92a5fefbf9345b9a03c059e1120f80f486ca42406d37d" Oct 30 16:44:04.575284 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.574174 4861 scope.go:122] "RemoveContainer" containerID="5961c9cfb09f31cacd458122e3cacecd72bdb5c0b3e5ca2f314eaa6acc15dbaa" Oct 30 16:44:04.576993 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:44:04.575555 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5961c9cfb09f31cacd458122e3cacecd72bdb5c0b3e5ca2f314eaa6acc15dbaa\": container with ID starting with 5961c9cfb09f31cacd458122e3cacecd72bdb5c0b3e5ca2f314eaa6acc15dbaa not found: ID does not exist" containerID="5961c9cfb09f31cacd458122e3cacecd72bdb5c0b3e5ca2f314eaa6acc15dbaa" Oct 30 16:44:04.576993 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.575597 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5961c9cfb09f31cacd458122e3cacecd72bdb5c0b3e5ca2f314eaa6acc15dbaa"} err="failed to get container status \"5961c9cfb09f31cacd458122e3cacecd72bdb5c0b3e5ca2f314eaa6acc15dbaa\": rpc error: code = NotFound desc = could not find container \"5961c9cfb09f31cacd458122e3cacecd72bdb5c0b3e5ca2f314eaa6acc15dbaa\": container with ID starting with 5961c9cfb09f31cacd458122e3cacecd72bdb5c0b3e5ca2f314eaa6acc15dbaa not found: ID does not exist" Oct 30 16:44:04.576993 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.575622 4861 scope.go:122] "RemoveContainer" containerID="5edecec21ad47c959bc92a5fefbf9345b9a03c059e1120f80f486ca42406d37d" Oct 30 16:44:04.576993 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:44:04.576889 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5edecec21ad47c959bc92a5fefbf9345b9a03c059e1120f80f486ca42406d37d\": container with ID starting with 5edecec21ad47c959bc92a5fefbf9345b9a03c059e1120f80f486ca42406d37d not found: ID does not exist" containerID="5edecec21ad47c959bc92a5fefbf9345b9a03c059e1120f80f486ca42406d37d" Oct 30 16:44:04.576993 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.576919 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5edecec21ad47c959bc92a5fefbf9345b9a03c059e1120f80f486ca42406d37d"} err="failed to get container status \"5edecec21ad47c959bc92a5fefbf9345b9a03c059e1120f80f486ca42406d37d\": rpc error: code = NotFound desc = could not find container \"5edecec21ad47c959bc92a5fefbf9345b9a03c059e1120f80f486ca42406d37d\": container with ID starting with 5edecec21ad47c959bc92a5fefbf9345b9a03c059e1120f80f486ca42406d37d not found: ID does not exist" Oct 30 16:44:04.831114 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:44:04.831082 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fecce1f_7675_40f7_9e2f_a849ceb4bc04.slice/crio-f65c2fbd753c672980a60ee8a04b35bd60fa7bb18367585b50a8c987e1c9d106 WatchSource:0}: Error finding container f65c2fbd753c672980a60ee8a04b35bd60fa7bb18367585b50a8c987e1c9d106: Status 404 returned error can't find the container with id f65c2fbd753c672980a60ee8a04b35bd60fa7bb18367585b50a8c987e1c9d106 Oct 30 16:44:04.831304 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.831248 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b"] Oct 30 16:44:04.870780 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.870754 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c"] Oct 30 16:44:04.924198 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:04.915839 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:05.287791 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:05.287757 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" event={"ID":"2fecce1f-7675-40f7-9e2f-a849ceb4bc04","Type":"ContainerStarted","Data":"ede57ab7013a046507ec776946bb89d2cb280e0fd64fa6b0e25cdb7a205d7c60"} Oct 30 16:44:05.287973 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:05.287816 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" event={"ID":"2fecce1f-7675-40f7-9e2f-a849ceb4bc04","Type":"ContainerStarted","Data":"f65c2fbd753c672980a60ee8a04b35bd60fa7bb18367585b50a8c987e1c9d106"} Oct 30 16:44:05.287973 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:05.287876 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" Oct 30 16:44:05.289540 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:05.289513 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" event={"ID":"ed177420-80fa-483f-a97a-62286e7f20fd","Type":"ContainerStarted","Data":"a32c8622372e6c6f3a50231a352ddbfb399f0f48ec979abf395f9a339c1719d2"} Oct 30 16:44:05.289665 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:05.289557 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" event={"ID":"ed177420-80fa-483f-a97a-62286e7f20fd","Type":"ContainerStarted","Data":"32b9a1322d3e1a5329e6b9d807da4a965538c2a56417a44cc5687a4e160ef585"} Oct 30 16:44:05.289665 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:05.289586 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" Oct 30 16:44:05.332990 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:05.332948 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" podStartSLOduration=2.332935892 podStartE2EDuration="2.332935892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:05.329453146 +0000 UTC m=+1275.672127198" watchObservedRunningTime="2026-10-30 16:44:05.332935892 +0000 UTC m=+1275.675609944" Oct 30 16:44:05.376893 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:05.376847 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" podStartSLOduration=2.376830184 podStartE2EDuration="2.376830184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:05.373693585 +0000 UTC m=+1275.716367623" watchObservedRunningTime="2026-10-30 16:44:05.376830184 +0000 UTC m=+1275.719504223" Oct 30 16:44:05.808327 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:05.808283 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ac300a-fa56-4c84-8c54-ff4d677e9555" path="/var/lib/kubelet/pods/38ac300a-fa56-4c84-8c54-ff4d677e9555/volumes" Oct 30 16:44:06.312485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:06.312189 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:06.467215 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:06.467168 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:07.108152 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:07.108004 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:07.571928 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:07.569975 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:07.963607 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:07.963514 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:08.344883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.344850 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h"] Oct 30 16:44:08.350357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.350334 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.372227 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.372197 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h"] Oct 30 16:44:08.378684 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.378658 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4a4c68-openshift-default-dockercfg-dddkb\"" Oct 30 16:44:08.430745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.430490 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-token\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.430745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.430592 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-podinfo\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.430745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.430641 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-data\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.431129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.430954 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-workload-socket\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.431129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.431011 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-workload-certs\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.431129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.431093 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bfc8bc-b483-458e-8580-b32c699fd17b-istiod-ca-cert\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.431320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.431173 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:08.431320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.431183 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-envoy\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.431320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.431239 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-credential-socket\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.431320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.431269 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4njp4\" (UniqueName: \"kubernetes.io/projected/b7bfc8bc-b483-458e-8580-b32c699fd17b-kube-api-access-4njp4\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.535749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.534798 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-workload-socket\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.535749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.534874 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-workload-certs\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.535749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.534924 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bfc8bc-b483-458e-8580-b32c699fd17b-istiod-ca-cert\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.535749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.535017 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-envoy\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.535749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.535061 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-credential-socket\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.535749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.535096 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4njp4\" (UniqueName: \"kubernetes.io/projected/b7bfc8bc-b483-458e-8580-b32c699fd17b-kube-api-access-4njp4\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.535749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.535160 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-token\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.535749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.535203 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-podinfo\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.535749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.535251 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-data\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.536488 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.535807 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-data\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.536488 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.536067 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-workload-socket\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.536488 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.536295 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-workload-certs\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.551769 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.536932 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bfc8bc-b483-458e-8580-b32c699fd17b-istiod-ca-cert\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.551769 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.540362 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-credential-socket\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.580754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.577713 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:08.603831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.600359 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-podinfo\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.607753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.603994 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-envoy\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.607753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.604200 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-token\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.607753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.605068 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4njp4\" (UniqueName: \"kubernetes.io/projected/b7bfc8bc-b483-458e-8580-b32c699fd17b-kube-api-access-4njp4\") pod \"e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.675752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.675004 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:08.900828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:08.900683 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:09.048020 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:09.047984 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:09.334648 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:09.334618 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h"] Oct 30 16:44:09.342358 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:44:09.342330 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7bfc8bc_b483_458e_8580_b32c699fd17b.slice/crio-590e1eda550c94bb30f13eb2c8ebfb21f2219e7756e6452f195d81148bf01a3a WatchSource:0}: Error finding container 590e1eda550c94bb30f13eb2c8ebfb21f2219e7756e6452f195d81148bf01a3a: Status 404 returned error can't find the container with id 590e1eda550c94bb30f13eb2c8ebfb21f2219e7756e6452f195d81148bf01a3a Oct 30 16:44:09.344834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:09.344801 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:09.344916 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:09.344874 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:09.344916 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:09.344909 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:10.232755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:10.221749 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:10.376812 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:10.376775 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" event={"ID":"b7bfc8bc-b483-458e-8580-b32c699fd17b","Type":"ContainerStarted","Data":"e1c0aba7496559f921f20473391d3f74e8c5ed5bab855fc9be864c74043cb965"} Oct 30 16:44:10.376998 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:10.376839 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" event={"ID":"b7bfc8bc-b483-458e-8580-b32c699fd17b","Type":"ContainerStarted","Data":"590e1eda550c94bb30f13eb2c8ebfb21f2219e7756e6452f195d81148bf01a3a"} Oct 30 16:44:10.458568 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:10.458522 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" podStartSLOduration=2.458510028 podStartE2EDuration="2.458510028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:10.445783733 +0000 UTC m=+1280.788457769" watchObservedRunningTime="2026-10-30 16:44:10.458510028 +0000 UTC m=+1280.801184066" Oct 30 16:44:10.677446 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:10.677364 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:10.692186 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:10.692153 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:11.383691 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:11.383665 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:11.384591 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:11.384573 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:44:12.254583 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:12.254538 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:12.952746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:12.948303 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:13.130749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:13.126505 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:14.693790 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:14.693739 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:15.134274 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:15.134234 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:15.292298 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:15.292265 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:15.609445 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:15.609383 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:16.301921 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:16.301893 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" Oct 30 16:44:16.302399 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:16.301961 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" Oct 30 16:44:16.729930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:16.729891 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:44:16.730264 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:16.730234 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" podUID="8ab63f0b-0a42-4361-ad08-58464c7e65ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:44:16.876622 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:16.876593 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:17.640412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:17.640384 4861 generic.go:356] "Generic (PLEG): container finished" podID="8ab63f0b-0a42-4361-ad08-58464c7e65ab" containerID="2a9faf03ba898ac4ce5e3f6d54207081d5c965ad22207174e7bb726cd70eeaef" exitCode=0 Oct 30 16:44:17.640794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:17.640451 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" event={"ID":"8ab63f0b-0a42-4361-ad08-58464c7e65ab","Type":"ContainerDied","Data":"2a9faf03ba898ac4ce5e3f6d54207081d5c965ad22207174e7bb726cd70eeaef"} Oct 30 16:44:17.640907 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:17.640879 4861 scope.go:122] "RemoveContainer" containerID="2a9faf03ba898ac4ce5e3f6d54207081d5c965ad22207174e7bb726cd70eeaef" Oct 30 16:44:17.642421 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:17.642394 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:17.642490 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:17.642464 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:17.642528 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:17.642495 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:18.688812 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:18.688777 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" event={"ID":"8ab63f0b-0a42-4361-ad08-58464c7e65ab","Type":"ContainerStarted","Data":"b547b5db367875176735f3983c209c5ff8f49489e4a7ef231fe0b81e6794f8f0"} Oct 30 16:44:19.030569 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:19.030536 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:19.625839 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:19.625806 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:44:19.626134 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:19.625869 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 16:44:19.626320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:19.626294 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" podUID="8ab63f0b-0a42-4361-ad08-58464c7e65ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 16:44:20.165561 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:20.165525 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:20.330503 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:20.330466 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:20.407762 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:44:20.407511 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:44:20.617023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:20.616409 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 16:44:20.617023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:20.616484 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" podUID="8ab63f0b-0a42-4361-ad08-58464c7e65ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 16:44:21.562234 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:21.562180 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:21.618786 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:21.615518 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 16:44:21.618786 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:21.615616 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" podUID="8ab63f0b-0a42-4361-ad08-58464c7e65ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 16:44:22.623748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:22.622319 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 16:44:22.623748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:22.622379 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" podUID="8ab63f0b-0a42-4361-ad08-58464c7e65ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 16:44:23.488019 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:23.487919 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:23.628593 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:23.628552 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 16:44:23.632478 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:23.632444 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" podUID="8ab63f0b-0a42-4361-ad08-58464c7e65ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 16:44:23.633104 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:23.632220 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:24.626996 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:24.626953 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:44:24.628766 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:24.628710 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:44:24.629548 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:24.629523 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:44:25.388535 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.388492 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn"] Oct 30 16:44:25.394787 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.394755 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:44:25.446077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.446041 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-08fcc8-serving-cert\"" Oct 30 16:44:25.452165 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.452137 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn"] Oct 30 16:44:25.495918 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.495887 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45769419-34ec-42d9-b5de-035877d01d2b-metrics-certs\") pod \"maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn\" (UID: \"45769419-34ec-42d9-b5de-035877d01d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:44:25.496069 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.495942 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs5cv\" (UniqueName: \"kubernetes.io/projected/45769419-34ec-42d9-b5de-035877d01d2b-kube-api-access-bs5cv\") pod \"maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn\" (UID: \"45769419-34ec-42d9-b5de-035877d01d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:44:25.496069 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.496057 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45769419-34ec-42d9-b5de-035877d01d2b-openshift-service-ca\") pod \"maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn\" (UID: \"45769419-34ec-42d9-b5de-035877d01d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:44:25.496154 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.496124 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45769419-34ec-42d9-b5de-035877d01d2b-maas-api-tls\") pod \"maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn\" (UID: \"45769419-34ec-42d9-b5de-035877d01d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:44:25.525453 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.525425 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:25.598742 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.598680 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45769419-34ec-42d9-b5de-035877d01d2b-openshift-service-ca\") pod \"maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn\" (UID: \"45769419-34ec-42d9-b5de-035877d01d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:44:25.599113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.599080 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45769419-34ec-42d9-b5de-035877d01d2b-maas-api-tls\") pod \"maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn\" (UID: \"45769419-34ec-42d9-b5de-035877d01d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:44:25.599611 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.599574 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45769419-34ec-42d9-b5de-035877d01d2b-metrics-certs\") pod \"maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn\" (UID: \"45769419-34ec-42d9-b5de-035877d01d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:44:25.616013 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.615818 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs5cv\" (UniqueName: \"kubernetes.io/projected/45769419-34ec-42d9-b5de-035877d01d2b-kube-api-access-bs5cv\") pod \"maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn\" (UID: \"45769419-34ec-42d9-b5de-035877d01d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:44:25.616360 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.614193 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45769419-34ec-42d9-b5de-035877d01d2b-openshift-service-ca\") pod \"maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn\" (UID: \"45769419-34ec-42d9-b5de-035877d01d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:44:25.629761 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.624416 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45769419-34ec-42d9-b5de-035877d01d2b-metrics-certs\") pod \"maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn\" (UID: \"45769419-34ec-42d9-b5de-035877d01d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:44:25.661368 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.661219 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45769419-34ec-42d9-b5de-035877d01d2b-maas-api-tls\") pod \"maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn\" (UID: \"45769419-34ec-42d9-b5de-035877d01d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:44:25.669396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.669362 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:25.710485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.710400 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs5cv\" (UniqueName: \"kubernetes.io/projected/45769419-34ec-42d9-b5de-035877d01d2b-kube-api-access-bs5cv\") pod \"maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn\" (UID: \"45769419-34ec-42d9-b5de-035877d01d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:44:25.752746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:25.752232 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:44:26.133518 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:26.132878 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn"] Oct 30 16:44:27.078228 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:27.078187 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" event={"ID":"45769419-34ec-42d9-b5de-035877d01d2b","Type":"ContainerStarted","Data":"139d755e9fa9fd8f15e5ca41f5cc1a6c6a383bb88d2dbd5d3b41dfcfbd16731e"} Oct 30 16:44:27.078842 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:27.078814 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" event={"ID":"45769419-34ec-42d9-b5de-035877d01d2b","Type":"ContainerStarted","Data":"b4c9b7f331f23149835f331bc62647b855ab264cea18df6d4664b981434125f5"} Oct 30 16:44:27.079032 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:27.079019 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:44:27.130387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:27.130322 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" podStartSLOduration=1.7651104279999998 podStartE2EDuration="2.130305887s" totalImagesPullingTime="365.195459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:27.127914469 +0000 UTC m=+1297.470588509" watchObservedRunningTime="2026-10-30 16:44:27.130305887 +0000 UTC m=+1297.472979924" Oct 30 16:44:27.579844 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:27.579809 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:28.645098 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:28.645056 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg"] Oct 30 16:44:28.645584 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:28.645305 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" podUID="64a4e246-be6d-4d34-92d2-7d69ff5fb454" containerName="istio-proxy" containerID="cri-o://ce2e704956cb7dc579d9f67211585ff7a08c3ee14895ee00bde3742939142747" gracePeriod=30 Oct 30 16:44:28.658917 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:28.658875 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw"] Oct 30 16:44:28.659328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:28.659295 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" podUID="8c6122cd-f566-44be-b180-ae3a2578ddfa" containerName="istio-proxy" containerID="cri-o://0e28be9e0f031e1c608721b40b004044c2d700643ea82e1877bc4a68ae610aac" gracePeriod=30 Oct 30 16:44:28.889575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:28.889538 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:29.069022 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.068555 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:29.251064 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.250982 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:29.581690 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.581658 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7"] Oct 30 16:44:29.586620 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.586598 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" Oct 30 16:44:29.615989 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.615927 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7"] Oct 30 16:44:29.622502 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.622337 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0a98d1\"" Oct 30 16:44:29.622502 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.622457 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0a98d1-dockercfg-xqdbd\"" Oct 30 16:44:29.719186 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.719150 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbttf\" (UniqueName: \"kubernetes.io/projected/2e07cfe3-baff-49e7-8bf7-73715fa20049-kube-api-access-zbttf\") pod \"payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7\" (UID: \"2e07cfe3-baff-49e7-8bf7-73715fa20049\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" Oct 30 16:44:29.719656 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.719368 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e07cfe3-baff-49e7-8bf7-73715fa20049-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7\" (UID: \"2e07cfe3-baff-49e7-8bf7-73715fa20049\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" Oct 30 16:44:29.730579 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.730551 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp"] Oct 30 16:44:29.730799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.730777 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" podUID="866b3779-6bd0-4c61-a9cf-96161b1dc6d3" containerName="istio-proxy" containerID="cri-o://a8f67ecc31218e46f1f5a36386ee022d16cd83367e2dfb8406968ac3ab77bc39" gracePeriod=30 Oct 30 16:44:29.734363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.734339 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65"] Oct 30 16:44:29.734549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.734525 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" podUID="4c3c5184-e45a-4301-af8f-85e8003a171a" containerName="istio-proxy" containerID="cri-o://e4c9c18baf95be70caec7c463170d00c6c2d54c159afb85ef004b831556b765e" gracePeriod=30 Oct 30 16:44:29.738219 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.738195 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm"] Oct 30 16:44:29.738433 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.738390 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" podUID="5a3cf99a-1f00-4e00-891c-01b06d04a4b0" containerName="istio-proxy" containerID="cri-o://fa259f36cbf581a56c6ce685b896ee9956b8e4d17eee61ff66b307486f7bc744" gracePeriod=30 Oct 30 16:44:29.743685 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.743663 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5"] Oct 30 16:44:29.743860 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.743840 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" podUID="ceeed7ea-7144-48e0-8d06-b0a10f26f50a" containerName="istio-proxy" containerID="cri-o://792f3c5a18ddce125f0d562a1ccab9b7a2c8444f19f4b078102ad2458f45b48a" gracePeriod=30 Oct 30 16:44:29.820549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.820508 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbttf\" (UniqueName: \"kubernetes.io/projected/2e07cfe3-baff-49e7-8bf7-73715fa20049-kube-api-access-zbttf\") pod \"payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7\" (UID: \"2e07cfe3-baff-49e7-8bf7-73715fa20049\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" Oct 30 16:44:29.820774 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.820713 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e07cfe3-baff-49e7-8bf7-73715fa20049-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7\" (UID: \"2e07cfe3-baff-49e7-8bf7-73715fa20049\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" Oct 30 16:44:29.821635 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.821607 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e07cfe3-baff-49e7-8bf7-73715fa20049-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7\" (UID: \"2e07cfe3-baff-49e7-8bf7-73715fa20049\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" Oct 30 16:44:29.845040 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.844973 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbttf\" (UniqueName: \"kubernetes.io/projected/2e07cfe3-baff-49e7-8bf7-73715fa20049-kube-api-access-zbttf\") pod \"payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7\" (UID: \"2e07cfe3-baff-49e7-8bf7-73715fa20049\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" Oct 30 16:44:29.922871 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:29.922834 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" Oct 30 16:44:30.238481 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:30.238447 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7"] Oct 30 16:44:30.238773 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:44:30.238746 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e07cfe3_baff_49e7_8bf7_73715fa20049.slice/crio-89ae89003adcd2c9941c858c4b3f9f7307301cac2c7b48bafcd8ea6dd21d61fd WatchSource:0}: Error finding container 89ae89003adcd2c9941c858c4b3f9f7307301cac2c7b48bafcd8ea6dd21d61fd: Status 404 returned error can't find the container with id 89ae89003adcd2c9941c858c4b3f9f7307301cac2c7b48bafcd8ea6dd21d61fd Oct 30 16:44:30.264417 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:30.264389 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:31.189281 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:31.189235 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" event={"ID":"2e07cfe3-baff-49e7-8bf7-73715fa20049","Type":"ContainerStarted","Data":"89ae89003adcd2c9941c858c4b3f9f7307301cac2c7b48bafcd8ea6dd21d61fd"} Oct 30 16:44:31.726090 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:31.726056 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:32.212752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.192910 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns"] Oct 30 16:44:32.212752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.198233 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" Oct 30 16:44:32.240746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.239250 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-843ab3\"" Oct 30 16:44:32.240746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.239385 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-843ab3-dockercfg-5jd66\"" Oct 30 16:44:32.324747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.312956 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e75dc21-662f-4048-8455-df78d3db3b9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns\" (UID: \"5e75dc21-662f-4048-8455-df78d3db3b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" Oct 30 16:44:32.324747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.313082 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5q8j\" (UniqueName: \"kubernetes.io/projected/5e75dc21-662f-4048-8455-df78d3db3b9a-kube-api-access-d5q8j\") pod \"payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns\" (UID: \"5e75dc21-662f-4048-8455-df78d3db3b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" Oct 30 16:44:32.356416 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.342366 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns"] Oct 30 16:44:32.402195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.400437 4861 generic.go:356] "Generic (PLEG): container finished" podID="e5026254-8aef-43a6-b079-072241e3b7b4" containerID="0b2315a01b98298ca15952da55947fe27d96b433e97250ac1632c8889c65e916" exitCode=0 Oct 30 16:44:32.402195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.400555 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" event={"ID":"e5026254-8aef-43a6-b079-072241e3b7b4","Type":"ContainerDied","Data":"0b2315a01b98298ca15952da55947fe27d96b433e97250ac1632c8889c65e916"} Oct 30 16:44:32.402195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.401278 4861 scope.go:122] "RemoveContainer" containerID="0b2315a01b98298ca15952da55947fe27d96b433e97250ac1632c8889c65e916" Oct 30 16:44:32.429536 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.411652 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:32.429536 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.411753 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:32.429536 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.411797 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:32.429536 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.414451 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e75dc21-662f-4048-8455-df78d3db3b9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns\" (UID: \"5e75dc21-662f-4048-8455-df78d3db3b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" Oct 30 16:44:32.429536 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.414500 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5q8j\" (UniqueName: \"kubernetes.io/projected/5e75dc21-662f-4048-8455-df78d3db3b9a-kube-api-access-d5q8j\") pod \"payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns\" (UID: \"5e75dc21-662f-4048-8455-df78d3db3b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" Oct 30 16:44:32.429536 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.415392 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e75dc21-662f-4048-8455-df78d3db3b9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns\" (UID: \"5e75dc21-662f-4048-8455-df78d3db3b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" Oct 30 16:44:32.442970 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.442946 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5q8j\" (UniqueName: \"kubernetes.io/projected/5e75dc21-662f-4048-8455-df78d3db3b9a-kube-api-access-d5q8j\") pod \"payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns\" (UID: \"5e75dc21-662f-4048-8455-df78d3db3b9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" Oct 30 16:44:32.589633 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.589612 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" Oct 30 16:44:32.911569 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:32.911543 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns"] Oct 30 16:44:32.912982 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:44:32.912949 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e75dc21_662f_4048_8455_df78d3db3b9a.slice/crio-d294bc08ae75f5645d88d9f5f63c8209987ee5b5bfb4f6f4177b7e2bec3417d3 WatchSource:0}: Error finding container d294bc08ae75f5645d88d9f5f63c8209987ee5b5bfb4f6f4177b7e2bec3417d3: Status 404 returned error can't find the container with id d294bc08ae75f5645d88d9f5f63c8209987ee5b5bfb4f6f4177b7e2bec3417d3 Oct 30 16:44:33.091599 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:33.091570 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:44:33.411884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:33.411849 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" event={"ID":"e5026254-8aef-43a6-b079-072241e3b7b4","Type":"ContainerStarted","Data":"08d46bbeff961651f9f4464d9419447c030091d00707bc2bbfd755ad9d381562"} Oct 30 16:44:33.413799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:33.413766 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" event={"ID":"5e75dc21-662f-4048-8455-df78d3db3b9a","Type":"ContainerStarted","Data":"04f3253cdeac22fda884e7560ce7f633be14abe64029d98076ccd688bd3a1509"} Oct 30 16:44:33.413947 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:33.413823 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" event={"ID":"5e75dc21-662f-4048-8455-df78d3db3b9a","Type":"ContainerStarted","Data":"d294bc08ae75f5645d88d9f5f63c8209987ee5b5bfb4f6f4177b7e2bec3417d3"} Oct 30 16:44:33.414158 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:33.414137 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" Oct 30 16:44:33.415643 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:33.415613 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" event={"ID":"2e07cfe3-baff-49e7-8bf7-73715fa20049","Type":"ContainerStarted","Data":"613ce6efe31f66a6cf3fa0e1445a1bae6880deb4441dca7565074986eb238d23"} Oct 30 16:44:33.415777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:33.415715 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" Oct 30 16:44:33.505542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:33.505489 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" podStartSLOduration=2.31475521 podStartE2EDuration="4.505473536s" totalImagesPullingTime="2.190718326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:33.503189118 +0000 UTC m=+1303.845863159" watchObservedRunningTime="2026-10-30 16:44:33.505473536 +0000 UTC m=+1303.848147575" Oct 30 16:44:33.545657 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:33.545504 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" podStartSLOduration=1.5454880389999999 podStartE2EDuration="1.545488039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:33.544528513 +0000 UTC m=+1303.887202555" watchObservedRunningTime="2026-10-30 16:44:33.545488039 +0000 UTC m=+1303.888162084" Oct 30 16:44:33.623432 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:33.623402 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:34.089401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.089370 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:44:34.162448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.162415 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-workload-certs\") pod \"8c6122cd-f566-44be-b180-ae3a2578ddfa\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " Oct 30 16:44:34.162831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.162809 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-workload-socket\") pod \"8c6122cd-f566-44be-b180-ae3a2578ddfa\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " Oct 30 16:44:34.163027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.163010 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-podinfo\") pod \"8c6122cd-f566-44be-b180-ae3a2578ddfa\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " Oct 30 16:44:34.163227 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.163208 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c6122cd-f566-44be-b180-ae3a2578ddfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6122cd-f566-44be-b180-ae3a2578ddfa-istiod-ca-cert\") pod \"8c6122cd-f566-44be-b180-ae3a2578ddfa\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " Oct 30 16:44:34.163352 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.163338 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-envoy\") pod \"8c6122cd-f566-44be-b180-ae3a2578ddfa\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " Oct 30 16:44:34.163501 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.163485 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-token\" (UniqueName: \"kubernetes.io/projected/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-token\") pod \"8c6122cd-f566-44be-b180-ae3a2578ddfa\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " Oct 30 16:44:34.163716 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.163700 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-credential-socket\") pod \"8c6122cd-f566-44be-b180-ae3a2578ddfa\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " Oct 30 16:44:34.163895 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.163880 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-data\") pod \"8c6122cd-f566-44be-b180-ae3a2578ddfa\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " Oct 30 16:44:34.164035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.164022 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6122cd-f566-44be-b180-ae3a2578ddfa-kube-api-access-cpg9t\" (UniqueName: \"kubernetes.io/projected/8c6122cd-f566-44be-b180-ae3a2578ddfa-kube-api-access-cpg9t\") pod \"8c6122cd-f566-44be-b180-ae3a2578ddfa\" (UID: \"8c6122cd-f566-44be-b180-ae3a2578ddfa\") " Oct 30 16:44:34.165608 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.165571 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-credential-socket" pod "8c6122cd-f566-44be-b180-ae3a2578ddfa" (UID: "8c6122cd-f566-44be-b180-ae3a2578ddfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:34.166018 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.165997 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-workload-certs" pod "8c6122cd-f566-44be-b180-ae3a2578ddfa" (UID: "8c6122cd-f566-44be-b180-ae3a2578ddfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:34.166202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.166169 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-data" pod "8c6122cd-f566-44be-b180-ae3a2578ddfa" (UID: "8c6122cd-f566-44be-b180-ae3a2578ddfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:34.166366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.166329 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-workload-socket" pod "8c6122cd-f566-44be-b180-ae3a2578ddfa" (UID: "8c6122cd-f566-44be-b180-ae3a2578ddfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:34.166851 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.166818 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6122cd-f566-44be-b180-ae3a2578ddfa-istiod-ca-cert" pod "8c6122cd-f566-44be-b180-ae3a2578ddfa" (UID: "8c6122cd-f566-44be-b180-ae3a2578ddfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:34.244681 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.225667 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-token" pod "8c6122cd-f566-44be-b180-ae3a2578ddfa" (UID: "8c6122cd-f566-44be-b180-ae3a2578ddfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:34.244681 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.225829 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6122cd-f566-44be-b180-ae3a2578ddfa-kube-api-access-cpg9t" pod "8c6122cd-f566-44be-b180-ae3a2578ddfa" (UID: "8c6122cd-f566-44be-b180-ae3a2578ddfa"). InnerVolumeSpecName "kube-api-access-cpg9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:34.244681 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.225883 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-envoy" pod "8c6122cd-f566-44be-b180-ae3a2578ddfa" (UID: "8c6122cd-f566-44be-b180-ae3a2578ddfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:34.264228 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.262488 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq"] Oct 30 16:44:34.264228 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.262754 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" podUID="dd2973e4-f250-465b-af94-f3fd93960586" containerName="istio-proxy" containerID="cri-o://76620f23a2f688708c2ce10d93e1a98e961a99b20358bdbf0ee79ef7acd1017b" gracePeriod=30 Oct 30 16:44:34.275794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.274924 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.275794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.274961 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6122cd-f566-44be-b180-ae3a2578ddfa-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.275794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.274985 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.275794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.274999 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.275794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.275013 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.275794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.275027 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.275794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.275042 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpg9t\" (UniqueName: \"kubernetes.io/projected/8c6122cd-f566-44be-b180-ae3a2578ddfa-kube-api-access-cpg9t\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.275794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.275055 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6122cd-f566-44be-b180-ae3a2578ddfa-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.359903 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.359876 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:44:34.372905 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.363001 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:44:34.372905 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.363067 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" podUID="e5026254-8aef-43a6-b079-072241e3b7b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 16:44:34.443805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.439746 4861 generic.go:356] "Generic (PLEG): container finished" podID="64a4e246-be6d-4d34-92d2-7d69ff5fb454" containerID="ce2e704956cb7dc579d9f67211585ff7a08c3ee14895ee00bde3742939142747" exitCode=0 Oct 30 16:44:34.445038 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.444632 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-podinfo" pod "8c6122cd-f566-44be-b180-ae3a2578ddfa" (UID: "8c6122cd-f566-44be-b180-ae3a2578ddfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:44:34.455757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.452280 4861 generic.go:356] "Generic (PLEG): container finished" podID="8c6122cd-f566-44be-b180-ae3a2578ddfa" containerID="0e28be9e0f031e1c608721b40b004044c2d700643ea82e1877bc4a68ae610aac" exitCode=0 Oct 30 16:44:34.455757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.453282 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" Oct 30 16:44:34.462193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.462137 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:44:34.467778 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.467758 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" event={"ID":"64a4e246-be6d-4d34-92d2-7d69ff5fb454","Type":"ContainerDied","Data":"ce2e704956cb7dc579d9f67211585ff7a08c3ee14895ee00bde3742939142747"} Oct 30 16:44:34.467933 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.467921 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p"] Oct 30 16:44:34.468068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.468056 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" event={"ID":"64a4e246-be6d-4d34-92d2-7d69ff5fb454","Type":"ContainerDied","Data":"cc62673fcf3c20ff86ebc7ac8ec9b72b50fcd0ec54d15b1d9bd1312c1ba16b2c"} Oct 30 16:44:34.468165 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.468155 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" event={"ID":"8c6122cd-f566-44be-b180-ae3a2578ddfa","Type":"ContainerDied","Data":"0e28be9e0f031e1c608721b40b004044c2d700643ea82e1877bc4a68ae610aac"} Oct 30 16:44:34.468246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.468237 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw" event={"ID":"8c6122cd-f566-44be-b180-ae3a2578ddfa","Type":"ContainerDied","Data":"e9994032e266c456b470347fc63bd2784f935bead9861ae7b628b706bdb4d88d"} Oct 30 16:44:34.468321 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.468313 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2"] Oct 30 16:44:34.468408 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.468400 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w"] Oct 30 16:44:34.468696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.468668 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" podUID="5ad8806d-d9cd-42f7-8507-a2f765165f43" containerName="istio-proxy" containerID="cri-o://a8c4a891042fe5e514c770c893232a33aff4964c193e3126a72cdd12215a0fab" gracePeriod=30 Oct 30 16:44:34.468977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.468940 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerName="istio-proxy" containerID="cri-o://0c53813042be3d532dc9097d09324cb8d7d811697ef5b163863be946af2795d7" gracePeriod=30 Oct 30 16:44:34.469247 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.467978 4861 scope.go:122] "RemoveContainer" containerID="ce2e704956cb7dc579d9f67211585ff7a08c3ee14895ee00bde3742939142747" Oct 30 16:44:34.470566 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.469223 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" containerName="istio-proxy" containerID="cri-o://db432519444c6d0b3cfe88986ad1bd600d42373e9802bf7e552e33d426cb2a3d" gracePeriod=30 Oct 30 16:44:34.482445 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.482415 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-podinfo\") pod \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " Oct 30 16:44:34.482576 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.482462 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-workload-socket\") pod \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " Oct 30 16:44:34.482576 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.482558 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-credential-socket\") pod \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " Oct 30 16:44:34.482677 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.482595 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a4e246-be6d-4d34-92d2-7d69ff5fb454-kube-api-access-jfrnb\" (UniqueName: \"kubernetes.io/projected/64a4e246-be6d-4d34-92d2-7d69ff5fb454-kube-api-access-jfrnb\") pod \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " Oct 30 16:44:34.482677 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.482638 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istiod-ca-cert\") pod \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " Oct 30 16:44:34.485312 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.483737 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-credential-socket" pod "64a4e246-be6d-4d34-92d2-7d69ff5fb454" (UID: "64a4e246-be6d-4d34-92d2-7d69ff5fb454"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:34.485312 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.484060 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istiod-ca-cert" pod "64a4e246-be6d-4d34-92d2-7d69ff5fb454" (UID: "64a4e246-be6d-4d34-92d2-7d69ff5fb454"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:34.485312 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.484130 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-workload-socket" pod "64a4e246-be6d-4d34-92d2-7d69ff5fb454" (UID: "64a4e246-be6d-4d34-92d2-7d69ff5fb454"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:34.485312 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.484237 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-envoy\") pod \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " Oct 30 16:44:34.485312 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.484329 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-data\") pod \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " Oct 30 16:44:34.485312 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.484406 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-token\" (UniqueName: \"kubernetes.io/projected/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-token\") pod \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " Oct 30 16:44:34.485312 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.484496 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-workload-certs\") pod \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\" (UID: \"64a4e246-be6d-4d34-92d2-7d69ff5fb454\") " Oct 30 16:44:34.486485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.485965 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.486485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.486045 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6122cd-f566-44be-b180-ae3a2578ddfa-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.486485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.486060 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.486485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.486074 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.486655 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.486552 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-workload-certs" pod "64a4e246-be6d-4d34-92d2-7d69ff5fb454" (UID: "64a4e246-be6d-4d34-92d2-7d69ff5fb454"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:34.486940 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.486886 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-data" pod "64a4e246-be6d-4d34-92d2-7d69ff5fb454" (UID: "64a4e246-be6d-4d34-92d2-7d69ff5fb454"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:34.487529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.487475 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-podinfo" pod "64a4e246-be6d-4d34-92d2-7d69ff5fb454" (UID: "64a4e246-be6d-4d34-92d2-7d69ff5fb454"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:44:34.487529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.487485 4861 scope.go:122] "RemoveContainer" containerID="ce2e704956cb7dc579d9f67211585ff7a08c3ee14895ee00bde3742939142747" Oct 30 16:44:34.487690 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.487612 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a4e246-be6d-4d34-92d2-7d69ff5fb454-kube-api-access-jfrnb" pod "64a4e246-be6d-4d34-92d2-7d69ff5fb454" (UID: "64a4e246-be6d-4d34-92d2-7d69ff5fb454"). InnerVolumeSpecName "kube-api-access-jfrnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:34.488024 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:44:34.487919 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2e704956cb7dc579d9f67211585ff7a08c3ee14895ee00bde3742939142747\": container with ID starting with ce2e704956cb7dc579d9f67211585ff7a08c3ee14895ee00bde3742939142747 not found: ID does not exist" containerID="ce2e704956cb7dc579d9f67211585ff7a08c3ee14895ee00bde3742939142747" Oct 30 16:44:34.488024 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.487942 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2e704956cb7dc579d9f67211585ff7a08c3ee14895ee00bde3742939142747"} err="failed to get container status \"ce2e704956cb7dc579d9f67211585ff7a08c3ee14895ee00bde3742939142747\": rpc error: code = NotFound desc = could not find container \"ce2e704956cb7dc579d9f67211585ff7a08c3ee14895ee00bde3742939142747\": container with ID starting with ce2e704956cb7dc579d9f67211585ff7a08c3ee14895ee00bde3742939142747 not found: ID does not exist" Oct 30 16:44:34.488024 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.487958 4861 scope.go:122] "RemoveContainer" containerID="0e28be9e0f031e1c608721b40b004044c2d700643ea82e1877bc4a68ae610aac" Oct 30 16:44:34.489653 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.489621 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-token" pod "64a4e246-be6d-4d34-92d2-7d69ff5fb454" (UID: "64a4e246-be6d-4d34-92d2-7d69ff5fb454"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:34.490518 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.490496 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-envoy" pod "64a4e246-be6d-4d34-92d2-7d69ff5fb454" (UID: "64a4e246-be6d-4d34-92d2-7d69ff5fb454"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:34.498840 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.498769 4861 scope.go:122] "RemoveContainer" containerID="0e28be9e0f031e1c608721b40b004044c2d700643ea82e1877bc4a68ae610aac" Oct 30 16:44:34.499125 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:44:34.499103 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e28be9e0f031e1c608721b40b004044c2d700643ea82e1877bc4a68ae610aac\": container with ID starting with 0e28be9e0f031e1c608721b40b004044c2d700643ea82e1877bc4a68ae610aac not found: ID does not exist" containerID="0e28be9e0f031e1c608721b40b004044c2d700643ea82e1877bc4a68ae610aac" Oct 30 16:44:34.499194 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.499129 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e28be9e0f031e1c608721b40b004044c2d700643ea82e1877bc4a68ae610aac"} err="failed to get container status \"0e28be9e0f031e1c608721b40b004044c2d700643ea82e1877bc4a68ae610aac\": rpc error: code = NotFound desc = could not find container \"0e28be9e0f031e1c608721b40b004044c2d700643ea82e1877bc4a68ae610aac\": container with ID starting with 0e28be9e0f031e1c608721b40b004044c2d700643ea82e1877bc4a68ae610aac not found: ID does not exist" Oct 30 16:44:34.587883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.587534 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfrnb\" (UniqueName: \"kubernetes.io/projected/64a4e246-be6d-4d34-92d2-7d69ff5fb454-kube-api-access-jfrnb\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.587883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.587562 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.587883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.587570 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.587883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.587579 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.587883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.587586 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64a4e246-be6d-4d34-92d2-7d69ff5fb454-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.587883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.587593 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64a4e246-be6d-4d34-92d2-7d69ff5fb454-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:34.644989 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.644842 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw"] Oct 30 16:44:34.649711 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:34.649682 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hmmdw"] Oct 30 16:44:35.327357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.327284 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:35.385471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.381428 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:44:35.385471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.381482 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" podUID="e5026254-8aef-43a6-b079-072241e3b7b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 16:44:35.410306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.399788 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:44:35.498303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.498279 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-podinfo\") pod \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " Oct 30 16:44:35.500353 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.500326 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-token\" (UniqueName: \"kubernetes.io/projected/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-token\") pod \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " Oct 30 16:44:35.500564 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.500545 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-envoy\") pod \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " Oct 30 16:44:35.500713 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.500697 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istiod-ca-cert\") pod \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " Oct 30 16:44:35.520955 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.501216 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:44:35.520955 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.503328 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-podinfo" pod "866b3779-6bd0-4c61-a9cf-96161b1dc6d3" (UID: "866b3779-6bd0-4c61-a9cf-96161b1dc6d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:44:35.520955 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.512035 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istiod-ca-cert" pod "866b3779-6bd0-4c61-a9cf-96161b1dc6d3" (UID: "866b3779-6bd0-4c61-a9cf-96161b1dc6d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:35.544658 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.543932 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:44:35.547840 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.547816 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-workload-certs\") pod \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " Oct 30 16:44:35.548112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.548096 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-credential-socket\") pod \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " Oct 30 16:44:35.548248 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.548235 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-kube-api-access-8zpfm\" (UniqueName: \"kubernetes.io/projected/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-kube-api-access-8zpfm\") pod \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " Oct 30 16:44:35.548446 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.548434 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-workload-socket\") pod \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " Oct 30 16:44:35.548602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.548590 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-data\") pod \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\" (UID: \"866b3779-6bd0-4c61-a9cf-96161b1dc6d3\") " Oct 30 16:44:35.549310 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.549292 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.549421 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.549409 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.549789 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.549715 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-credential-socket" pod "866b3779-6bd0-4c61-a9cf-96161b1dc6d3" (UID: "866b3779-6bd0-4c61-a9cf-96161b1dc6d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.550056 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.550037 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-workload-socket" pod "866b3779-6bd0-4c61-a9cf-96161b1dc6d3" (UID: "866b3779-6bd0-4c61-a9cf-96161b1dc6d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.550246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.550227 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-workload-certs" pod "866b3779-6bd0-4c61-a9cf-96161b1dc6d3" (UID: "866b3779-6bd0-4c61-a9cf-96161b1dc6d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.550455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.550418 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-data" pod "866b3779-6bd0-4c61-a9cf-96161b1dc6d3" (UID: "866b3779-6bd0-4c61-a9cf-96161b1dc6d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.553174 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.553135 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-token" pod "866b3779-6bd0-4c61-a9cf-96161b1dc6d3" (UID: "866b3779-6bd0-4c61-a9cf-96161b1dc6d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:35.558089 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.558054 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-kube-api-access-8zpfm" pod "866b3779-6bd0-4c61-a9cf-96161b1dc6d3" (UID: "866b3779-6bd0-4c61-a9cf-96161b1dc6d3"). InnerVolumeSpecName "kube-api-access-8zpfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:35.558716 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.558683 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-envoy" pod "866b3779-6bd0-4c61-a9cf-96161b1dc6d3" (UID: "866b3779-6bd0-4c61-a9cf-96161b1dc6d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.560510 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.560488 4861 generic.go:356] "Generic (PLEG): container finished" podID="866b3779-6bd0-4c61-a9cf-96161b1dc6d3" containerID="a8f67ecc31218e46f1f5a36386ee022d16cd83367e2dfb8406968ac3ab77bc39" exitCode=0 Oct 30 16:44:35.560868 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.560847 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" event={"ID":"866b3779-6bd0-4c61-a9cf-96161b1dc6d3","Type":"ContainerDied","Data":"a8f67ecc31218e46f1f5a36386ee022d16cd83367e2dfb8406968ac3ab77bc39"} Oct 30 16:44:35.561024 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.561008 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" event={"ID":"866b3779-6bd0-4c61-a9cf-96161b1dc6d3","Type":"ContainerDied","Data":"bce4b55aa9eb5f033b55f0ff3ba76487b943883e2cc6e21cfd127a5bf23326fe"} Oct 30 16:44:35.561132 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.561121 4861 scope.go:122] "RemoveContainer" containerID="a8f67ecc31218e46f1f5a36386ee022d16cd83367e2dfb8406968ac3ab77bc39" Oct 30 16:44:35.561370 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.561355 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp" Oct 30 16:44:35.564935 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.564561 4861 generic.go:356] "Generic (PLEG): container finished" podID="4c3c5184-e45a-4301-af8f-85e8003a171a" containerID="e4c9c18baf95be70caec7c463170d00c6c2d54c159afb85ef004b831556b765e" exitCode=0 Oct 30 16:44:35.564935 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.564671 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" event={"ID":"4c3c5184-e45a-4301-af8f-85e8003a171a","Type":"ContainerDied","Data":"e4c9c18baf95be70caec7c463170d00c6c2d54c159afb85ef004b831556b765e"} Oct 30 16:44:35.565078 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.564711 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" event={"ID":"4c3c5184-e45a-4301-af8f-85e8003a171a","Type":"ContainerDied","Data":"098683ba16268aabbbb3ad0fdfb76a2846a8b0277ec0a5512cf1951c585ab74b"} Oct 30 16:44:35.565206 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.565190 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65" Oct 30 16:44:35.567258 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.567231 4861 generic.go:356] "Generic (PLEG): container finished" podID="5a3cf99a-1f00-4e00-891c-01b06d04a4b0" containerID="fa259f36cbf581a56c6ce685b896ee9956b8e4d17eee61ff66b307486f7bc744" exitCode=0 Oct 30 16:44:35.567519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.567308 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" event={"ID":"5a3cf99a-1f00-4e00-891c-01b06d04a4b0","Type":"ContainerDied","Data":"fa259f36cbf581a56c6ce685b896ee9956b8e4d17eee61ff66b307486f7bc744"} Oct 30 16:44:35.567519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.567343 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" event={"ID":"5a3cf99a-1f00-4e00-891c-01b06d04a4b0","Type":"ContainerDied","Data":"9ed7268cf18fc4ed98b6ebc2abf29602b3b2c655d24326912579d6d84d10a15d"} Oct 30 16:44:35.567519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.567403 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" Oct 30 16:44:35.573613 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.572280 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg" Oct 30 16:44:35.600559 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.600535 4861 scope.go:122] "RemoveContainer" containerID="a8f67ecc31218e46f1f5a36386ee022d16cd83367e2dfb8406968ac3ab77bc39" Oct 30 16:44:35.610383 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:44:35.610341 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f67ecc31218e46f1f5a36386ee022d16cd83367e2dfb8406968ac3ab77bc39\": container with ID starting with a8f67ecc31218e46f1f5a36386ee022d16cd83367e2dfb8406968ac3ab77bc39 not found: ID does not exist" containerID="a8f67ecc31218e46f1f5a36386ee022d16cd83367e2dfb8406968ac3ab77bc39" Oct 30 16:44:35.610611 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.610578 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f67ecc31218e46f1f5a36386ee022d16cd83367e2dfb8406968ac3ab77bc39"} err="failed to get container status \"a8f67ecc31218e46f1f5a36386ee022d16cd83367e2dfb8406968ac3ab77bc39\": rpc error: code = NotFound desc = could not find container \"a8f67ecc31218e46f1f5a36386ee022d16cd83367e2dfb8406968ac3ab77bc39\": container with ID starting with a8f67ecc31218e46f1f5a36386ee022d16cd83367e2dfb8406968ac3ab77bc39 not found: ID does not exist" Oct 30 16:44:35.610773 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.610758 4861 scope.go:122] "RemoveContainer" containerID="e4c9c18baf95be70caec7c463170d00c6c2d54c159afb85ef004b831556b765e" Oct 30 16:44:35.618922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.615479 4861 generic.go:356] "Generic (PLEG): container finished" podID="ceeed7ea-7144-48e0-8d06-b0a10f26f50a" containerID="792f3c5a18ddce125f0d562a1ccab9b7a2c8444f19f4b078102ad2458f45b48a" exitCode=0 Oct 30 16:44:35.619097 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.615504 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" event={"ID":"ceeed7ea-7144-48e0-8d06-b0a10f26f50a","Type":"ContainerDied","Data":"792f3c5a18ddce125f0d562a1ccab9b7a2c8444f19f4b078102ad2458f45b48a"} Oct 30 16:44:35.653065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.653036 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c3c5184-e45a-4301-af8f-85e8003a171a-kube-api-access-gbxts\" (UniqueName: \"kubernetes.io/projected/4c3c5184-e45a-4301-af8f-85e8003a171a-kube-api-access-gbxts\") pod \"4c3c5184-e45a-4301-af8f-85e8003a171a\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " Oct 30 16:44:35.653279 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.653266 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-envoy\") pod \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " Oct 30 16:44:35.653360 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.653351 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-token\" (UniqueName: \"kubernetes.io/projected/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-token\") pod \"4c3c5184-e45a-4301-af8f-85e8003a171a\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " Oct 30 16:44:35.653430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.653422 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-workload-socket\") pod \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " Oct 30 16:44:35.653530 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.653521 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-podinfo\") pod \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " Oct 30 16:44:35.653595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.653587 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-data\") pod \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " Oct 30 16:44:35.653665 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.653657 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c3c5184-e45a-4301-af8f-85e8003a171a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c3c5184-e45a-4301-af8f-85e8003a171a-istiod-ca-cert\") pod \"4c3c5184-e45a-4301-af8f-85e8003a171a\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " Oct 30 16:44:35.653839 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.653814 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-podinfo\") pod \"4c3c5184-e45a-4301-af8f-85e8003a171a\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " Oct 30 16:44:35.653940 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.653913 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-envoy\") pod \"4c3c5184-e45a-4301-af8f-85e8003a171a\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " Oct 30 16:44:35.653996 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.653945 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-workload-certs\") pod \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " Oct 30 16:44:35.654063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.654044 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-credential-socket\") pod \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " Oct 30 16:44:35.654131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.654116 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-token\" (UniqueName: \"kubernetes.io/projected/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-token\") pod \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " Oct 30 16:44:35.654199 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.654183 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-workload-socket\") pod \"4c3c5184-e45a-4301-af8f-85e8003a171a\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " Oct 30 16:44:35.654256 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.654227 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-data\") pod \"4c3c5184-e45a-4301-af8f-85e8003a171a\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " Oct 30 16:44:35.654307 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.654267 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-kube-api-access-vzpp5\" (UniqueName: \"kubernetes.io/projected/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-kube-api-access-vzpp5\") pod \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " Oct 30 16:44:35.654387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.654360 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istiod-ca-cert\") pod \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\" (UID: \"5a3cf99a-1f00-4e00-891c-01b06d04a4b0\") " Oct 30 16:44:35.654446 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.654405 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-credential-socket\") pod \"4c3c5184-e45a-4301-af8f-85e8003a171a\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " Oct 30 16:44:35.654499 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.654450 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-workload-certs\") pod \"4c3c5184-e45a-4301-af8f-85e8003a171a\" (UID: \"4c3c5184-e45a-4301-af8f-85e8003a171a\") " Oct 30 16:44:35.655451 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.655422 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.655559 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.655455 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.655559 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.655469 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.655559 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.655488 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.655559 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.655503 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.655559 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.655517 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zpfm\" (UniqueName: \"kubernetes.io/projected/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-kube-api-access-8zpfm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.655559 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.655530 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/866b3779-6bd0-4c61-a9cf-96161b1dc6d3-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.657339 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.657286 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-workload-certs" pod "4c3c5184-e45a-4301-af8f-85e8003a171a" (UID: "4c3c5184-e45a-4301-af8f-85e8003a171a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.657437 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.657343 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-workload-socket" pod "4c3c5184-e45a-4301-af8f-85e8003a171a" (UID: "4c3c5184-e45a-4301-af8f-85e8003a171a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.657531 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.657504 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-data" pod "4c3c5184-e45a-4301-af8f-85e8003a171a" (UID: "4c3c5184-e45a-4301-af8f-85e8003a171a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.659931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.659904 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istiod-ca-cert" pod "5a3cf99a-1f00-4e00-891c-01b06d04a4b0" (UID: "5a3cf99a-1f00-4e00-891c-01b06d04a4b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:35.660379 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.660353 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-credential-socket" pod "4c3c5184-e45a-4301-af8f-85e8003a171a" (UID: "4c3c5184-e45a-4301-af8f-85e8003a171a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.662486 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.662455 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-podinfo" pod "4c3c5184-e45a-4301-af8f-85e8003a171a" (UID: "4c3c5184-e45a-4301-af8f-85e8003a171a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:44:35.663687 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.663304 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-kube-api-access-vzpp5" pod "5a3cf99a-1f00-4e00-891c-01b06d04a4b0" (UID: "5a3cf99a-1f00-4e00-891c-01b06d04a4b0"). InnerVolumeSpecName "kube-api-access-vzpp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:35.663687 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.663462 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-credential-socket" pod "5a3cf99a-1f00-4e00-891c-01b06d04a4b0" (UID: "5a3cf99a-1f00-4e00-891c-01b06d04a4b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.664033 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.664005 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-token" pod "4c3c5184-e45a-4301-af8f-85e8003a171a" (UID: "4c3c5184-e45a-4301-af8f-85e8003a171a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:35.664887 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.664852 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-workload-certs" pod "5a3cf99a-1f00-4e00-891c-01b06d04a4b0" (UID: "5a3cf99a-1f00-4e00-891c-01b06d04a4b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.665862 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.665582 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c3c5184-e45a-4301-af8f-85e8003a171a-kube-api-access-gbxts" pod "4c3c5184-e45a-4301-af8f-85e8003a171a" (UID: "4c3c5184-e45a-4301-af8f-85e8003a171a"). InnerVolumeSpecName "kube-api-access-gbxts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:35.666331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.666033 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-workload-socket" pod "5a3cf99a-1f00-4e00-891c-01b06d04a4b0" (UID: "5a3cf99a-1f00-4e00-891c-01b06d04a4b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.666682 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.666659 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-data" pod "5a3cf99a-1f00-4e00-891c-01b06d04a4b0" (UID: "5a3cf99a-1f00-4e00-891c-01b06d04a4b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.667216 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.667054 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c3c5184-e45a-4301-af8f-85e8003a171a-istiod-ca-cert" pod "4c3c5184-e45a-4301-af8f-85e8003a171a" (UID: "4c3c5184-e45a-4301-af8f-85e8003a171a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:35.667216 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.667105 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-envoy" pod "5a3cf99a-1f00-4e00-891c-01b06d04a4b0" (UID: "5a3cf99a-1f00-4e00-891c-01b06d04a4b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.667838 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.667796 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-podinfo" pod "5a3cf99a-1f00-4e00-891c-01b06d04a4b0" (UID: "5a3cf99a-1f00-4e00-891c-01b06d04a4b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:44:35.669036 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.668976 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-token" pod "5a3cf99a-1f00-4e00-891c-01b06d04a4b0" (UID: "5a3cf99a-1f00-4e00-891c-01b06d04a4b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:35.677849 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.677818 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-envoy" pod "4c3c5184-e45a-4301-af8f-85e8003a171a" (UID: "4c3c5184-e45a-4301-af8f-85e8003a171a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.681012 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.680995 4861 scope.go:122] "RemoveContainer" containerID="e4c9c18baf95be70caec7c463170d00c6c2d54c159afb85ef004b831556b765e" Oct 30 16:44:35.681526 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:44:35.681500 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c9c18baf95be70caec7c463170d00c6c2d54c159afb85ef004b831556b765e\": container with ID starting with e4c9c18baf95be70caec7c463170d00c6c2d54c159afb85ef004b831556b765e not found: ID does not exist" containerID="e4c9c18baf95be70caec7c463170d00c6c2d54c159afb85ef004b831556b765e" Oct 30 16:44:35.681609 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.681531 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c9c18baf95be70caec7c463170d00c6c2d54c159afb85ef004b831556b765e"} err="failed to get container status \"e4c9c18baf95be70caec7c463170d00c6c2d54c159afb85ef004b831556b765e\": rpc error: code = NotFound desc = could not find container \"e4c9c18baf95be70caec7c463170d00c6c2d54c159afb85ef004b831556b765e\": container with ID starting with e4c9c18baf95be70caec7c463170d00c6c2d54c159afb85ef004b831556b765e not found: ID does not exist" Oct 30 16:44:35.681609 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.681546 4861 scope.go:122] "RemoveContainer" containerID="fa259f36cbf581a56c6ce685b896ee9956b8e4d17eee61ff66b307486f7bc744" Oct 30 16:44:35.685940 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.685921 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:44:35.691447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.691430 4861 scope.go:122] "RemoveContainer" containerID="fa259f36cbf581a56c6ce685b896ee9956b8e4d17eee61ff66b307486f7bc744" Oct 30 16:44:35.691748 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:44:35.691706 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa259f36cbf581a56c6ce685b896ee9956b8e4d17eee61ff66b307486f7bc744\": container with ID starting with fa259f36cbf581a56c6ce685b896ee9956b8e4d17eee61ff66b307486f7bc744 not found: ID does not exist" containerID="fa259f36cbf581a56c6ce685b896ee9956b8e4d17eee61ff66b307486f7bc744" Oct 30 16:44:35.691854 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.691748 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa259f36cbf581a56c6ce685b896ee9956b8e4d17eee61ff66b307486f7bc744"} err="failed to get container status \"fa259f36cbf581a56c6ce685b896ee9956b8e4d17eee61ff66b307486f7bc744\": rpc error: code = NotFound desc = could not find container \"fa259f36cbf581a56c6ce685b896ee9956b8e4d17eee61ff66b307486f7bc744\": container with ID starting with fa259f36cbf581a56c6ce685b896ee9956b8e4d17eee61ff66b307486f7bc744 not found: ID does not exist" Oct 30 16:44:35.758258 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.758178 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-podinfo\") pod \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " Oct 30 16:44:35.758258 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.758235 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-kube-api-access-mvxtg\" (UniqueName: \"kubernetes.io/projected/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-kube-api-access-mvxtg\") pod \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " Oct 30 16:44:35.758438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.758275 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-data\") pod \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " Oct 30 16:44:35.758438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.758332 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-credential-socket\") pod \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " Oct 30 16:44:35.758438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.758381 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istiod-ca-cert\") pod \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " Oct 30 16:44:35.758534 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.758451 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-token\" (UniqueName: \"kubernetes.io/projected/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-token\") pod \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " Oct 30 16:44:35.758534 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.758484 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-workload-socket\") pod \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " Oct 30 16:44:35.759274 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.758679 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-envoy\") pod \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " Oct 30 16:44:35.759274 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.758710 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-workload-certs\") pod \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\" (UID: \"ceeed7ea-7144-48e0-8d06-b0a10f26f50a\") " Oct 30 16:44:35.759274 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.758763 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-credential-socket" pod "ceeed7ea-7144-48e0-8d06-b0a10f26f50a" (UID: "ceeed7ea-7144-48e0-8d06-b0a10f26f50a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.759274 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.758886 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istiod-ca-cert" pod "ceeed7ea-7144-48e0-8d06-b0a10f26f50a" (UID: "ceeed7ea-7144-48e0-8d06-b0a10f26f50a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:35.759274 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.759074 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-workload-socket" pod "ceeed7ea-7144-48e0-8d06-b0a10f26f50a" (UID: "ceeed7ea-7144-48e0-8d06-b0a10f26f50a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.759274 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.759149 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-data" pod "ceeed7ea-7144-48e0-8d06-b0a10f26f50a" (UID: "ceeed7ea-7144-48e0-8d06-b0a10f26f50a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.759922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.759383 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-workload-certs" pod "ceeed7ea-7144-48e0-8d06-b0a10f26f50a" (UID: "ceeed7ea-7144-48e0-8d06-b0a10f26f50a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761021 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbxts\" (UniqueName: \"kubernetes.io/projected/4c3c5184-e45a-4301-af8f-85e8003a171a-kube-api-access-gbxts\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761050 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761064 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761076 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761089 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761102 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761114 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761126 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761137 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761150 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761163 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c3c5184-e45a-4301-af8f-85e8003a171a-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761175 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761187 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761200 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761211 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761222 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761230 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761237 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761245 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761253 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzpp5\" (UniqueName: \"kubernetes.io/projected/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-kube-api-access-vzpp5\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761264 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3cf99a-1f00-4e00-891c-01b06d04a4b0-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761273 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.761300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761281 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c3c5184-e45a-4301-af8f-85e8003a171a-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.762299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.761600 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-envoy" pod "ceeed7ea-7144-48e0-8d06-b0a10f26f50a" (UID: "ceeed7ea-7144-48e0-8d06-b0a10f26f50a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:35.762360 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.762311 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-kube-api-access-mvxtg" pod "ceeed7ea-7144-48e0-8d06-b0a10f26f50a" (UID: "ceeed7ea-7144-48e0-8d06-b0a10f26f50a"). InnerVolumeSpecName "kube-api-access-mvxtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:35.763485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.763454 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-token" pod "ceeed7ea-7144-48e0-8d06-b0a10f26f50a" (UID: "ceeed7ea-7144-48e0-8d06-b0a10f26f50a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:35.765050 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.765029 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-podinfo" pod "ceeed7ea-7144-48e0-8d06-b0a10f26f50a" (UID: "ceeed7ea-7144-48e0-8d06-b0a10f26f50a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:44:35.767998 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.767976 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg"] Oct 30 16:44:35.771564 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.771543 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-hghcg"] Oct 30 16:44:35.847260 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.847126 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a4e246-be6d-4d34-92d2-7d69ff5fb454" path="/var/lib/kubelet/pods/64a4e246-be6d-4d34-92d2-7d69ff5fb454/volumes" Oct 30 16:44:35.847794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.847775 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c6122cd-f566-44be-b180-ae3a2578ddfa" path="/var/lib/kubelet/pods/8c6122cd-f566-44be-b180-ae3a2578ddfa/volumes" Oct 30 16:44:35.862361 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.862250 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.862361 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.862273 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.862361 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.862284 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvxtg\" (UniqueName: \"kubernetes.io/projected/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-kube-api-access-mvxtg\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.862361 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.862294 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceeed7ea-7144-48e0-8d06-b0a10f26f50a-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:35.870841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.870820 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp"] Oct 30 16:44:35.874258 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.874240 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-mncsp"] Oct 30 16:44:35.950110 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.950070 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65"] Oct 30 16:44:35.956749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.956708 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-vnr65"] Oct 30 16:44:35.972364 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.971896 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm"] Oct 30 16:44:35.975794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:35.975775 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm"] Oct 30 16:44:36.319826 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:36.319280 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:44:36.319826 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:36.319343 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" podUID="e5026254-8aef-43a6-b079-072241e3b7b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 16:44:36.394711 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:36.394677 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:44:36.394883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:36.394756 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pdrwm" podUID="5a3cf99a-1f00-4e00-891c-01b06d04a4b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:44:36.641146 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:36.641068 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" Oct 30 16:44:36.641146 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:36.641118 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5" event={"ID":"ceeed7ea-7144-48e0-8d06-b0a10f26f50a","Type":"ContainerDied","Data":"249f35e5694bef1304168bac35533ad250ed94b6535658bbabeec7de314e7aed"} Oct 30 16:44:36.641665 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:36.641185 4861 scope.go:122] "RemoveContainer" containerID="792f3c5a18ddce125f0d562a1ccab9b7a2c8444f19f4b078102ad2458f45b48a" Oct 30 16:44:36.695833 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:36.695811 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5"] Oct 30 16:44:36.699885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:36.699857 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-l4hr5"] Oct 30 16:44:37.055614 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.055578 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh"] Oct 30 16:44:37.056243 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056227 4861 cpu_manager.go:414] "containerMap: removing container" podUID="4c3c5184-e45a-4301-af8f-85e8003a171a" containerName="istio-proxy" Oct 30 16:44:37.056289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056245 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c3c5184-e45a-4301-af8f-85e8003a171a" containerName="istio-proxy" Oct 30 16:44:37.056289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056280 4861 cpu_manager.go:414] "containerMap: removing container" podUID="5a3cf99a-1f00-4e00-891c-01b06d04a4b0" containerName="istio-proxy" Oct 30 16:44:37.056289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056285 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3cf99a-1f00-4e00-891c-01b06d04a4b0" containerName="istio-proxy" Oct 30 16:44:37.056375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056296 4861 cpu_manager.go:414] "containerMap: removing container" podUID="866b3779-6bd0-4c61-a9cf-96161b1dc6d3" containerName="istio-proxy" Oct 30 16:44:37.056375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056302 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="866b3779-6bd0-4c61-a9cf-96161b1dc6d3" containerName="istio-proxy" Oct 30 16:44:37.056375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056310 4861 cpu_manager.go:414] "containerMap: removing container" podUID="64a4e246-be6d-4d34-92d2-7d69ff5fb454" containerName="istio-proxy" Oct 30 16:44:37.056375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056314 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a4e246-be6d-4d34-92d2-7d69ff5fb454" containerName="istio-proxy" Oct 30 16:44:37.056375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056326 4861 cpu_manager.go:414] "containerMap: removing container" podUID="8c6122cd-f566-44be-b180-ae3a2578ddfa" containerName="istio-proxy" Oct 30 16:44:37.056375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056330 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c6122cd-f566-44be-b180-ae3a2578ddfa" containerName="istio-proxy" Oct 30 16:44:37.056375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056342 4861 cpu_manager.go:414] "containerMap: removing container" podUID="ceeed7ea-7144-48e0-8d06-b0a10f26f50a" containerName="istio-proxy" Oct 30 16:44:37.056375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056347 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceeed7ea-7144-48e0-8d06-b0a10f26f50a" containerName="istio-proxy" Oct 30 16:44:37.056595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056428 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="866b3779-6bd0-4c61-a9cf-96161b1dc6d3" containerName="istio-proxy" Oct 30 16:44:37.056595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056437 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a4e246-be6d-4d34-92d2-7d69ff5fb454" containerName="istio-proxy" Oct 30 16:44:37.056595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056443 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c6122cd-f566-44be-b180-ae3a2578ddfa" containerName="istio-proxy" Oct 30 16:44:37.056595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056451 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceeed7ea-7144-48e0-8d06-b0a10f26f50a" containerName="istio-proxy" Oct 30 16:44:37.056595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056457 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c3c5184-e45a-4301-af8f-85e8003a171a" containerName="istio-proxy" Oct 30 16:44:37.056595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.056466 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3cf99a-1f00-4e00-891c-01b06d04a4b0" containerName="istio-proxy" Oct 30 16:44:37.060172 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.060142 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:44:37.077681 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.077205 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4a4c68-serving-cert\"" Oct 30 16:44:37.081188 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.081166 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh"] Oct 30 16:44:37.180036 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.180007 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d78l\" (UniqueName: \"kubernetes.io/projected/9f783738-9f60-476b-a06c-90f25eda1d4f-kube-api-access-4d78l\") pod \"maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh\" (UID: \"9f783738-9f60-476b-a06c-90f25eda1d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:44:37.180240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.180219 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f783738-9f60-476b-a06c-90f25eda1d4f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh\" (UID: \"9f783738-9f60-476b-a06c-90f25eda1d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:44:37.180289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.180272 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f783738-9f60-476b-a06c-90f25eda1d4f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh\" (UID: \"9f783738-9f60-476b-a06c-90f25eda1d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:44:37.180383 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.180371 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f783738-9f60-476b-a06c-90f25eda1d4f-metrics-certs\") pod \"maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh\" (UID: \"9f783738-9f60-476b-a06c-90f25eda1d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:44:37.185994 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.185972 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:37.282338 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.282305 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f783738-9f60-476b-a06c-90f25eda1d4f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh\" (UID: \"9f783738-9f60-476b-a06c-90f25eda1d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:44:37.282523 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.282380 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f783738-9f60-476b-a06c-90f25eda1d4f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh\" (UID: \"9f783738-9f60-476b-a06c-90f25eda1d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:44:37.282523 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.282458 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f783738-9f60-476b-a06c-90f25eda1d4f-metrics-certs\") pod \"maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh\" (UID: \"9f783738-9f60-476b-a06c-90f25eda1d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:44:37.282642 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.282517 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d78l\" (UniqueName: \"kubernetes.io/projected/9f783738-9f60-476b-a06c-90f25eda1d4f-kube-api-access-4d78l\") pod \"maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh\" (UID: \"9f783738-9f60-476b-a06c-90f25eda1d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:44:37.283093 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.283071 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f783738-9f60-476b-a06c-90f25eda1d4f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh\" (UID: \"9f783738-9f60-476b-a06c-90f25eda1d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:44:37.285098 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.285078 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f783738-9f60-476b-a06c-90f25eda1d4f-metrics-certs\") pod \"maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh\" (UID: \"9f783738-9f60-476b-a06c-90f25eda1d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:44:37.285341 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.285319 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f783738-9f60-476b-a06c-90f25eda1d4f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh\" (UID: \"9f783738-9f60-476b-a06c-90f25eda1d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:44:37.328536 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.318111 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:44:37.328536 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.318170 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" podUID="e5026254-8aef-43a6-b079-072241e3b7b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 16:44:37.328536 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.328393 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d78l\" (UniqueName: \"kubernetes.io/projected/9f783738-9f60-476b-a06c-90f25eda1d4f-kube-api-access-4d78l\") pod \"maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh\" (UID: \"9f783738-9f60-476b-a06c-90f25eda1d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:44:37.332008 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.331978 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:37.400780 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.379181 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:44:37.531549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.531509 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:37.800246 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:44:37.800219 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f783738_9f60_476b_a06c_90f25eda1d4f.slice/crio-1206e83ba51d48c32c306909a63787e40cc44d2255cd9a806d63947d41d98c0b WatchSource:0}: Error finding container 1206e83ba51d48c32c306909a63787e40cc44d2255cd9a806d63947d41d98c0b: Status 404 returned error can't find the container with id 1206e83ba51d48c32c306909a63787e40cc44d2255cd9a806d63947d41d98c0b Oct 30 16:44:37.830926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.830898 4861 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:44:37.834954 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.834931 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c3c5184-e45a-4301-af8f-85e8003a171a" path="/var/lib/kubelet/pods/4c3c5184-e45a-4301-af8f-85e8003a171a/volumes" Oct 30 16:44:37.835498 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.835483 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a3cf99a-1f00-4e00-891c-01b06d04a4b0" path="/var/lib/kubelet/pods/5a3cf99a-1f00-4e00-891c-01b06d04a4b0/volumes" Oct 30 16:44:37.836033 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.836020 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="866b3779-6bd0-4c61-a9cf-96161b1dc6d3" path="/var/lib/kubelet/pods/866b3779-6bd0-4c61-a9cf-96161b1dc6d3/volumes" Oct 30 16:44:37.836870 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.836856 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceeed7ea-7144-48e0-8d06-b0a10f26f50a" path="/var/lib/kubelet/pods/ceeed7ea-7144-48e0-8d06-b0a10f26f50a/volumes" Oct 30 16:44:37.837195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.837182 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh"] Oct 30 16:44:37.838194 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:37.838179 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:44:38.310756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:38.310702 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:44:38.311021 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:38.310987 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" podUID="e5026254-8aef-43a6-b079-072241e3b7b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 16:44:38.663366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:38.663332 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" event={"ID":"9f783738-9f60-476b-a06c-90f25eda1d4f","Type":"ContainerStarted","Data":"7c52f7b67e036af42b517477212a1181c0fce491e283914659bd83726f040a53"} Oct 30 16:44:38.663560 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:38.663424 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" event={"ID":"9f783738-9f60-476b-a06c-90f25eda1d4f","Type":"ContainerStarted","Data":"1206e83ba51d48c32c306909a63787e40cc44d2255cd9a806d63947d41d98c0b"} Oct 30 16:44:38.663560 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:38.663453 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:44:39.310287 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.310251 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 16:44:39.310931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.310896 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" podUID="e5026254-8aef-43a6-b079-072241e3b7b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 16:44:39.423750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.419371 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:39.693902 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.693429 4861 generic.go:356] "Generic (PLEG): container finished" podID="dd2973e4-f250-465b-af94-f3fd93960586" containerID="76620f23a2f688708c2ce10d93e1a98e961a99b20358bdbf0ee79ef7acd1017b" exitCode=0 Oct 30 16:44:39.693902 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.693549 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" event={"ID":"dd2973e4-f250-465b-af94-f3fd93960586","Type":"ContainerDied","Data":"76620f23a2f688708c2ce10d93e1a98e961a99b20358bdbf0ee79ef7acd1017b"} Oct 30 16:44:39.695762 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.695600 4861 generic.go:356] "Generic (PLEG): container finished" podID="262816dd-5373-4ee6-8624-727fd8b649c8" containerID="db432519444c6d0b3cfe88986ad1bd600d42373e9802bf7e552e33d426cb2a3d" exitCode=0 Oct 30 16:44:39.695762 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.695686 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" event={"ID":"262816dd-5373-4ee6-8624-727fd8b649c8","Type":"ContainerDied","Data":"db432519444c6d0b3cfe88986ad1bd600d42373e9802bf7e552e33d426cb2a3d"} Oct 30 16:44:39.698289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.698267 4861 generic.go:356] "Generic (PLEG): container finished" podID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerID="0c53813042be3d532dc9097d09324cb8d7d811697ef5b163863be946af2795d7" exitCode=0 Oct 30 16:44:39.699249 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.698467 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" event={"ID":"b9db1487-cac4-489d-b0fb-e63b12a1703b","Type":"ContainerDied","Data":"0c53813042be3d532dc9097d09324cb8d7d811697ef5b163863be946af2795d7"} Oct 30 16:44:39.701661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.701001 4861 generic.go:356] "Generic (PLEG): container finished" podID="5ad8806d-d9cd-42f7-8507-a2f765165f43" containerID="a8c4a891042fe5e514c770c893232a33aff4964c193e3126a72cdd12215a0fab" exitCode=0 Oct 30 16:44:39.701661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.701612 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" event={"ID":"5ad8806d-d9cd-42f7-8507-a2f765165f43","Type":"ContainerDied","Data":"a8c4a891042fe5e514c770c893232a33aff4964c193e3126a72cdd12215a0fab"} Oct 30 16:44:39.796052 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.796021 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:44:39.864157 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.864041 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" podStartSLOduration=2.451618716 podStartE2EDuration="2.864020598s" totalImagesPullingTime="412.401882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:38.727015559 +0000 UTC m=+1309.069689598" watchObservedRunningTime="2026-10-30 16:44:39.864020598 +0000 UTC m=+1310.206694637" Oct 30 16:44:39.898477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.895412 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd2973e4-f250-465b-af94-f3fd93960586-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2973e4-f250-465b-af94-f3fd93960586-istio-podinfo\") pod \"dd2973e4-f250-465b-af94-f3fd93960586\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " Oct 30 16:44:39.898477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.895477 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-workload-certs\") pod \"dd2973e4-f250-465b-af94-f3fd93960586\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " Oct 30 16:44:39.898477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.895553 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2973e4-f250-465b-af94-f3fd93960586-kube-api-access-8mxxm\" (UniqueName: \"kubernetes.io/projected/dd2973e4-f250-465b-af94-f3fd93960586-kube-api-access-8mxxm\") pod \"dd2973e4-f250-465b-af94-f3fd93960586\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " Oct 30 16:44:39.898477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.895667 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-credential-socket\") pod \"dd2973e4-f250-465b-af94-f3fd93960586\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " Oct 30 16:44:39.898477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.895750 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-workload-socket\") pod \"dd2973e4-f250-465b-af94-f3fd93960586\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " Oct 30 16:44:39.898477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.895805 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd2973e4-f250-465b-af94-f3fd93960586-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2973e4-f250-465b-af94-f3fd93960586-istiod-ca-cert\") pod \"dd2973e4-f250-465b-af94-f3fd93960586\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " Oct 30 16:44:39.898477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.895839 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-istio-envoy\") pod \"dd2973e4-f250-465b-af94-f3fd93960586\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " Oct 30 16:44:39.898477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.895914 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd2973e4-f250-465b-af94-f3fd93960586-istio-token\" (UniqueName: \"kubernetes.io/projected/dd2973e4-f250-465b-af94-f3fd93960586-istio-token\") pod \"dd2973e4-f250-465b-af94-f3fd93960586\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " Oct 30 16:44:39.898477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.895943 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-istio-data\") pod \"dd2973e4-f250-465b-af94-f3fd93960586\" (UID: \"dd2973e4-f250-465b-af94-f3fd93960586\") " Oct 30 16:44:39.899104 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.898798 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-workload-socket" pod "dd2973e4-f250-465b-af94-f3fd93960586" (UID: "dd2973e4-f250-465b-af94-f3fd93960586"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:39.899161 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.899130 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2973e4-f250-465b-af94-f3fd93960586-istiod-ca-cert" pod "dd2973e4-f250-465b-af94-f3fd93960586" (UID: "dd2973e4-f250-465b-af94-f3fd93960586"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:39.899344 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.899324 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-istio-data" pod "dd2973e4-f250-465b-af94-f3fd93960586" (UID: "dd2973e4-f250-465b-af94-f3fd93960586"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:39.899860 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.899806 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-credential-socket" pod "dd2973e4-f250-465b-af94-f3fd93960586" (UID: "dd2973e4-f250-465b-af94-f3fd93960586"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:39.899973 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.899884 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-workload-certs" pod "dd2973e4-f250-465b-af94-f3fd93960586" (UID: "dd2973e4-f250-465b-af94-f3fd93960586"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:39.904205 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.904152 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2973e4-f250-465b-af94-f3fd93960586-kube-api-access-8mxxm" pod "dd2973e4-f250-465b-af94-f3fd93960586" (UID: "dd2973e4-f250-465b-af94-f3fd93960586"). InnerVolumeSpecName "kube-api-access-8mxxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:39.906141 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.906110 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd2973e4-f250-465b-af94-f3fd93960586-istio-podinfo" pod "dd2973e4-f250-465b-af94-f3fd93960586" (UID: "dd2973e4-f250-465b-af94-f3fd93960586"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:44:39.910094 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.910068 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-istio-envoy" pod "dd2973e4-f250-465b-af94-f3fd93960586" (UID: "dd2973e4-f250-465b-af94-f3fd93960586"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:39.910298 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.910262 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd2973e4-f250-465b-af94-f3fd93960586-istio-token" pod "dd2973e4-f250-465b-af94-f3fd93960586" (UID: "dd2973e4-f250-465b-af94-f3fd93960586"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:39.942749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.933975 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:44:39.945067 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.945036 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:44:39.954968 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.954934 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:44:39.998290 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.998259 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd2973e4-f250-465b-af94-f3fd93960586-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:39.998290 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.998286 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:39.998507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.998301 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mxxm\" (UniqueName: \"kubernetes.io/projected/dd2973e4-f250-465b-af94-f3fd93960586-kube-api-access-8mxxm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:39.998507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.998317 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:39.998507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.998327 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:39.998507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.998335 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd2973e4-f250-465b-af94-f3fd93960586-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:39.998507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.998344 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:39.998507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.998352 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd2973e4-f250-465b-af94-f3fd93960586-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:39.998507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:39.998359 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd2973e4-f250-465b-af94-f3fd93960586-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.099492 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.099458 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-workload-socket\") pod \"b9db1487-cac4-489d-b0fb-e63b12a1703b\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " Oct 30 16:44:40.099680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.099553 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-credential-socket\") pod \"5ad8806d-d9cd-42f7-8507-a2f765165f43\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " Oct 30 16:44:40.099680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.099601 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-workload-socket\") pod \"5ad8806d-d9cd-42f7-8507-a2f765165f43\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " Oct 30 16:44:40.099680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.099640 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-token\" (UniqueName: \"kubernetes.io/projected/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-token\") pod \"b9db1487-cac4-489d-b0fb-e63b12a1703b\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " Oct 30 16:44:40.099680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.099668 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-envoy\") pod \"b9db1487-cac4-489d-b0fb-e63b12a1703b\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " Oct 30 16:44:40.099931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.099705 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-workload-certs\") pod \"5ad8806d-d9cd-42f7-8507-a2f765165f43\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " Oct 30 16:44:40.099931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.099780 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-data\") pod \"5ad8806d-d9cd-42f7-8507-a2f765165f43\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " Oct 30 16:44:40.099931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.099779 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-workload-socket" pod "b9db1487-cac4-489d-b0fb-e63b12a1703b" (UID: "b9db1487-cac4-489d-b0fb-e63b12a1703b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:40.099931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.099821 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-workload-certs\") pod \"262816dd-5373-4ee6-8624-727fd8b649c8\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " Oct 30 16:44:40.099931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.099911 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-istio-envoy\") pod \"262816dd-5373-4ee6-8624-727fd8b649c8\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " Oct 30 16:44:40.100197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.099957 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-workload-certs\") pod \"b9db1487-cac4-489d-b0fb-e63b12a1703b\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " Oct 30 16:44:40.100197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.099857 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-credential-socket" pod "5ad8806d-d9cd-42f7-8507-a2f765165f43" (UID: "5ad8806d-d9cd-42f7-8507-a2f765165f43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:40.100197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100002 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-workload-socket\") pod \"262816dd-5373-4ee6-8624-727fd8b649c8\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " Oct 30 16:44:40.100197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.099995 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-workload-socket" pod "5ad8806d-d9cd-42f7-8507-a2f765165f43" (UID: "5ad8806d-d9cd-42f7-8507-a2f765165f43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:40.100197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100059 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-credential-socket\") pod \"b9db1487-cac4-489d-b0fb-e63b12a1703b\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " Oct 30 16:44:40.100197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100060 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-workload-certs" pod "5ad8806d-d9cd-42f7-8507-a2f765165f43" (UID: "5ad8806d-d9cd-42f7-8507-a2f765165f43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:40.100197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100119 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-envoy\") pod \"5ad8806d-d9cd-42f7-8507-a2f765165f43\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " Oct 30 16:44:40.100197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100158 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-credential-socket\") pod \"262816dd-5373-4ee6-8624-727fd8b649c8\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " Oct 30 16:44:40.100590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100229 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9db1487-cac4-489d-b0fb-e63b12a1703b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9db1487-cac4-489d-b0fb-e63b12a1703b-istiod-ca-cert\") pod \"b9db1487-cac4-489d-b0fb-e63b12a1703b\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " Oct 30 16:44:40.100590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100262 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-istio-data\") pod \"262816dd-5373-4ee6-8624-727fd8b649c8\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " Oct 30 16:44:40.100590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100270 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-workload-socket" pod "262816dd-5373-4ee6-8624-727fd8b649c8" (UID: "262816dd-5373-4ee6-8624-727fd8b649c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:40.100590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100315 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262816dd-5373-4ee6-8624-727fd8b649c8-istio-token\" (UniqueName: \"kubernetes.io/projected/262816dd-5373-4ee6-8624-727fd8b649c8-istio-token\") pod \"262816dd-5373-4ee6-8624-727fd8b649c8\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " Oct 30 16:44:40.100590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100352 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-workload-certs" pod "262816dd-5373-4ee6-8624-727fd8b649c8" (UID: "262816dd-5373-4ee6-8624-727fd8b649c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:40.100590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100359 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ad8806d-d9cd-42f7-8507-a2f765165f43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad8806d-d9cd-42f7-8507-a2f765165f43-istiod-ca-cert\") pod \"5ad8806d-d9cd-42f7-8507-a2f765165f43\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " Oct 30 16:44:40.100590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100425 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9db1487-cac4-489d-b0fb-e63b12a1703b-kube-api-access-rj9c8\" (UniqueName: \"kubernetes.io/projected/b9db1487-cac4-489d-b0fb-e63b12a1703b-kube-api-access-rj9c8\") pod \"b9db1487-cac4-489d-b0fb-e63b12a1703b\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " Oct 30 16:44:40.100590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100459 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-token\" (UniqueName: \"kubernetes.io/projected/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-token\") pod \"5ad8806d-d9cd-42f7-8507-a2f765165f43\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " Oct 30 16:44:40.100590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100521 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-data\") pod \"b9db1487-cac4-489d-b0fb-e63b12a1703b\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " Oct 30 16:44:40.100590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100564 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-workload-certs" pod "b9db1487-cac4-489d-b0fb-e63b12a1703b" (UID: "b9db1487-cac4-489d-b0fb-e63b12a1703b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:40.100590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100579 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262816dd-5373-4ee6-8624-727fd8b649c8-kube-api-access-482tf\" (UniqueName: \"kubernetes.io/projected/262816dd-5373-4ee6-8624-727fd8b649c8-kube-api-access-482tf\") pod \"262816dd-5373-4ee6-8624-727fd8b649c8\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " Oct 30 16:44:40.101152 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100625 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/262816dd-5373-4ee6-8624-727fd8b649c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262816dd-5373-4ee6-8624-727fd8b649c8-istio-podinfo\") pod \"262816dd-5373-4ee6-8624-727fd8b649c8\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " Oct 30 16:44:40.101152 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100638 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ad8806d-d9cd-42f7-8507-a2f765165f43-istiod-ca-cert" pod "5ad8806d-d9cd-42f7-8507-a2f765165f43" (UID: "5ad8806d-d9cd-42f7-8507-a2f765165f43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:40.101152 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100652 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/262816dd-5373-4ee6-8624-727fd8b649c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262816dd-5373-4ee6-8624-727fd8b649c8-istiod-ca-cert\") pod \"262816dd-5373-4ee6-8624-727fd8b649c8\" (UID: \"262816dd-5373-4ee6-8624-727fd8b649c8\") " Oct 30 16:44:40.101152 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100703 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-podinfo\") pod \"b9db1487-cac4-489d-b0fb-e63b12a1703b\" (UID: \"b9db1487-cac4-489d-b0fb-e63b12a1703b\") " Oct 30 16:44:40.101152 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100788 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-podinfo\") pod \"5ad8806d-d9cd-42f7-8507-a2f765165f43\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " Oct 30 16:44:40.101152 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100822 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ad8806d-d9cd-42f7-8507-a2f765165f43-kube-api-access-ffqth\" (UniqueName: \"kubernetes.io/projected/5ad8806d-d9cd-42f7-8507-a2f765165f43-kube-api-access-ffqth\") pod \"5ad8806d-d9cd-42f7-8507-a2f765165f43\" (UID: \"5ad8806d-d9cd-42f7-8507-a2f765165f43\") " Oct 30 16:44:40.101152 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.100881 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9db1487-cac4-489d-b0fb-e63b12a1703b-istiod-ca-cert" pod "b9db1487-cac4-489d-b0fb-e63b12a1703b" (UID: "b9db1487-cac4-489d-b0fb-e63b12a1703b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:40.132246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.102170 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-credential-socket" pod "262816dd-5373-4ee6-8624-727fd8b649c8" (UID: "262816dd-5373-4ee6-8624-727fd8b649c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:40.132246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.102343 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-credential-socket" pod "b9db1487-cac4-489d-b0fb-e63b12a1703b" (UID: "b9db1487-cac4-489d-b0fb-e63b12a1703b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:40.132246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.102771 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-data" pod "5ad8806d-d9cd-42f7-8507-a2f765165f43" (UID: "5ad8806d-d9cd-42f7-8507-a2f765165f43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:40.132246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.114000 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-istio-data" pod "262816dd-5373-4ee6-8624-727fd8b649c8" (UID: "262816dd-5373-4ee6-8624-727fd8b649c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:40.132246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.116350 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-data" pod "b9db1487-cac4-489d-b0fb-e63b12a1703b" (UID: "b9db1487-cac4-489d-b0fb-e63b12a1703b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:40.132246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.116675 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262816dd-5373-4ee6-8624-727fd8b649c8-istiod-ca-cert" pod "262816dd-5373-4ee6-8624-727fd8b649c8" (UID: "262816dd-5373-4ee6-8624-727fd8b649c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:40.132246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.117104 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-envoy" pod "5ad8806d-d9cd-42f7-8507-a2f765165f43" (UID: "5ad8806d-d9cd-42f7-8507-a2f765165f43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:40.133243 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133203 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.133375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133245 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9db1487-cac4-489d-b0fb-e63b12a1703b-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.133375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133261 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.133375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133208 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-podinfo" pod "b9db1487-cac4-489d-b0fb-e63b12a1703b" (UID: "b9db1487-cac4-489d-b0fb-e63b12a1703b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:44:40.133375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133274 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ad8806d-d9cd-42f7-8507-a2f765165f43-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.133375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133318 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.133375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133334 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262816dd-5373-4ee6-8624-727fd8b649c8-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.133375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133349 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.133375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133363 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.133798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133381 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.133798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133396 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.133798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133411 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.133798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133424 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.133798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133438 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.133798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133451 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.133798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133467 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.133798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.133480 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.134698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.134659 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad8806d-d9cd-42f7-8507-a2f765165f43-kube-api-access-ffqth" pod "5ad8806d-d9cd-42f7-8507-a2f765165f43" (UID: "5ad8806d-d9cd-42f7-8507-a2f765165f43"). InnerVolumeSpecName "kube-api-access-ffqth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:40.134968 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.134941 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-podinfo" pod "5ad8806d-d9cd-42f7-8507-a2f765165f43" (UID: "5ad8806d-d9cd-42f7-8507-a2f765165f43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:44:40.137320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.137210 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9db1487-cac4-489d-b0fb-e63b12a1703b-kube-api-access-rj9c8" pod "b9db1487-cac4-489d-b0fb-e63b12a1703b" (UID: "b9db1487-cac4-489d-b0fb-e63b12a1703b"). InnerVolumeSpecName "kube-api-access-rj9c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:40.137320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.137296 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262816dd-5373-4ee6-8624-727fd8b649c8-kube-api-access-482tf" pod "262816dd-5373-4ee6-8624-727fd8b649c8" (UID: "262816dd-5373-4ee6-8624-727fd8b649c8"). InnerVolumeSpecName "kube-api-access-482tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:40.138542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.137884 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-token" pod "5ad8806d-d9cd-42f7-8507-a2f765165f43" (UID: "5ad8806d-d9cd-42f7-8507-a2f765165f43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:40.138542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.137964 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-token" pod "b9db1487-cac4-489d-b0fb-e63b12a1703b" (UID: "b9db1487-cac4-489d-b0fb-e63b12a1703b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:40.138542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.137980 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-envoy" pod "b9db1487-cac4-489d-b0fb-e63b12a1703b" (UID: "b9db1487-cac4-489d-b0fb-e63b12a1703b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:40.138542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.138015 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-istio-envoy" pod "262816dd-5373-4ee6-8624-727fd8b649c8" (UID: "262816dd-5373-4ee6-8624-727fd8b649c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:44:40.139093 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.139061 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262816dd-5373-4ee6-8624-727fd8b649c8-istio-token" pod "262816dd-5373-4ee6-8624-727fd8b649c8" (UID: "262816dd-5373-4ee6-8624-727fd8b649c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:40.139833 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.139811 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/262816dd-5373-4ee6-8624-727fd8b649c8-istio-podinfo" pod "262816dd-5373-4ee6-8624-727fd8b649c8" (UID: "262816dd-5373-4ee6-8624-727fd8b649c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:44:40.235132 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.235097 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262816dd-5373-4ee6-8624-727fd8b649c8-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.235132 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.235129 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj9c8\" (UniqueName: \"kubernetes.io/projected/b9db1487-cac4-489d-b0fb-e63b12a1703b-kube-api-access-rj9c8\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.235132 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.235141 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.235407 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.235152 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-482tf\" (UniqueName: \"kubernetes.io/projected/262816dd-5373-4ee6-8624-727fd8b649c8-kube-api-access-482tf\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.235407 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.235160 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262816dd-5373-4ee6-8624-727fd8b649c8-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.235407 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.235169 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.235407 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.235177 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ad8806d-d9cd-42f7-8507-a2f765165f43-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.235407 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.235185 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffqth\" (UniqueName: \"kubernetes.io/projected/5ad8806d-d9cd-42f7-8507-a2f765165f43-kube-api-access-ffqth\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.235407 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.235193 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.235407 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.235201 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9db1487-cac4-489d-b0fb-e63b12a1703b-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.235407 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.235208 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262816dd-5373-4ee6-8624-727fd8b649c8-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:40.329246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.329211 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:44:40.330389 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.330364 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:44:40.733831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.733798 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" Oct 30 16:44:40.733831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.733811 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq" event={"ID":"dd2973e4-f250-465b-af94-f3fd93960586","Type":"ContainerDied","Data":"848786dc07de0661b385ebc82a278607dac33db67ea08b70aa1180566eccb1f0"} Oct 30 16:44:40.734092 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.733870 4861 scope.go:122] "RemoveContainer" containerID="76620f23a2f688708c2ce10d93e1a98e961a99b20358bdbf0ee79ef7acd1017b" Oct 30 16:44:40.735462 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.735341 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" Oct 30 16:44:40.735462 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.735358 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2" event={"ID":"262816dd-5373-4ee6-8624-727fd8b649c8","Type":"ContainerDied","Data":"ca15d1db81cac60498df72bd402e88e24a28765443502fab957b959002ac6c54"} Oct 30 16:44:40.736983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.736967 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" Oct 30 16:44:40.737090 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.737020 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p" event={"ID":"b9db1487-cac4-489d-b0fb-e63b12a1703b","Type":"ContainerDied","Data":"b4d1fcb1368491306ee502bd7a9cbeb6504426b37aa5b59be6efe75b65fcd1c0"} Oct 30 16:44:40.738777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.738750 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" event={"ID":"5ad8806d-d9cd-42f7-8507-a2f765165f43","Type":"ContainerDied","Data":"b7d0d5503901b1ec04d606a2c3d43b253d1a4dd0c1f2e7d462515a57229ecf69"} Oct 30 16:44:40.738909 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.738751 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w" Oct 30 16:44:40.747782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.747675 4861 scope.go:122] "RemoveContainer" containerID="db432519444c6d0b3cfe88986ad1bd600d42373e9802bf7e552e33d426cb2a3d" Oct 30 16:44:40.757947 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.757929 4861 scope.go:122] "RemoveContainer" containerID="0c53813042be3d532dc9097d09324cb8d7d811697ef5b163863be946af2795d7" Oct 30 16:44:40.772908 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.772888 4861 scope.go:122] "RemoveContainer" containerID="a8c4a891042fe5e514c770c893232a33aff4964c193e3126a72cdd12215a0fab" Oct 30 16:44:40.849048 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.849017 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq"] Oct 30 16:44:40.855983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.855953 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-qrwnq"] Oct 30 16:44:40.890421 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.890399 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w"] Oct 30 16:44:40.894662 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.894641 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7xn9w"] Oct 30 16:44:40.940972 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.940942 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2"] Oct 30 16:44:40.948261 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.948236 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-w8qg2"] Oct 30 16:44:40.982988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.982953 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p"] Oct 30 16:44:40.987209 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:40.987183 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-7c98p"] Oct 30 16:44:41.383526 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:41.383490 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:41.832190 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:41.832157 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" path="/var/lib/kubelet/pods/262816dd-5373-4ee6-8624-727fd8b649c8/volumes" Oct 30 16:44:41.832640 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:41.832626 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ad8806d-d9cd-42f7-8507-a2f765165f43" path="/var/lib/kubelet/pods/5ad8806d-d9cd-42f7-8507-a2f765165f43/volumes" Oct 30 16:44:41.833078 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:41.833065 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" path="/var/lib/kubelet/pods/b9db1487-cac4-489d-b0fb-e63b12a1703b/volumes" Oct 30 16:44:41.833828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:41.833812 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd2973e4-f250-465b-af94-f3fd93960586" path="/var/lib/kubelet/pods/dd2973e4-f250-465b-af94-f3fd93960586/volumes" Oct 30 16:44:42.726422 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:42.726387 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:42.897746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:42.897652 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:43.204802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.204628 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:43.236745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.235794 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8"] Oct 30 16:44:43.239168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.237066 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerName="istio-proxy" Oct 30 16:44:43.239168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.237091 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerName="istio-proxy" Oct 30 16:44:43.239168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.237107 4861 cpu_manager.go:414] "containerMap: removing container" podUID="5ad8806d-d9cd-42f7-8507-a2f765165f43" containerName="istio-proxy" Oct 30 16:44:43.239168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.237128 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ad8806d-d9cd-42f7-8507-a2f765165f43" containerName="istio-proxy" Oct 30 16:44:43.239168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.237148 4861 cpu_manager.go:414] "containerMap: removing container" podUID="dd2973e4-f250-465b-af94-f3fd93960586" containerName="istio-proxy" Oct 30 16:44:43.239168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.237156 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd2973e4-f250-465b-af94-f3fd93960586" containerName="istio-proxy" Oct 30 16:44:43.239168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.237169 4861 cpu_manager.go:414] "containerMap: removing container" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" containerName="istio-proxy" Oct 30 16:44:43.239168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.237178 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" containerName="istio-proxy" Oct 30 16:44:43.239168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.237614 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9db1487-cac4-489d-b0fb-e63b12a1703b" containerName="istio-proxy" Oct 30 16:44:43.239168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.237644 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ad8806d-d9cd-42f7-8507-a2f765165f43" containerName="istio-proxy" Oct 30 16:44:43.239168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.237667 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd2973e4-f250-465b-af94-f3fd93960586" containerName="istio-proxy" Oct 30 16:44:43.239168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.237707 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="262816dd-5373-4ee6-8624-727fd8b649c8" containerName="istio-proxy" Oct 30 16:44:43.270708 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.266931 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8"] Oct 30 16:44:43.270708 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.267074 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" Oct 30 16:44:43.291770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.291110 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7c23a4\"" Oct 30 16:44:43.309852 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.304993 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7c23a4-dockercfg-jnzm9\"" Oct 30 16:44:43.354805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.354716 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba88b21-2400-499d-960d-03320e5b9dcd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8\" (UID: \"4ba88b21-2400-499d-960d-03320e5b9dcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" Oct 30 16:44:43.355007 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.354918 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6fc\" (UniqueName: \"kubernetes.io/projected/4ba88b21-2400-499d-960d-03320e5b9dcd-kube-api-access-fr6fc\") pod \"payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8\" (UID: \"4ba88b21-2400-499d-960d-03320e5b9dcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" Oct 30 16:44:43.433605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.433562 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9"] Oct 30 16:44:43.438898 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.438879 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" Oct 30 16:44:43.445295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.445264 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9"] Oct 30 16:44:43.455734 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.455653 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba88b21-2400-499d-960d-03320e5b9dcd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8\" (UID: \"4ba88b21-2400-499d-960d-03320e5b9dcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" Oct 30 16:44:43.455843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.455790 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6fc\" (UniqueName: \"kubernetes.io/projected/4ba88b21-2400-499d-960d-03320e5b9dcd-kube-api-access-fr6fc\") pod \"payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8\" (UID: \"4ba88b21-2400-499d-960d-03320e5b9dcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" Oct 30 16:44:43.456322 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.456301 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba88b21-2400-499d-960d-03320e5b9dcd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8\" (UID: \"4ba88b21-2400-499d-960d-03320e5b9dcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" Oct 30 16:44:43.480469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.480440 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6fc\" (UniqueName: \"kubernetes.io/projected/4ba88b21-2400-499d-960d-03320e5b9dcd-kube-api-access-fr6fc\") pod \"payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8\" (UID: \"4ba88b21-2400-499d-960d-03320e5b9dcd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" Oct 30 16:44:43.557422 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.557391 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9fefa6-cb76-453e-b5c2-37e429f2cd5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9\" (UID: \"df9fefa6-cb76-453e-b5c2-37e429f2cd5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" Oct 30 16:44:43.557553 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.557529 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zq2w\" (UniqueName: \"kubernetes.io/projected/df9fefa6-cb76-453e-b5c2-37e429f2cd5d-kube-api-access-7zq2w\") pod \"payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9\" (UID: \"df9fefa6-cb76-453e-b5c2-37e429f2cd5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" Oct 30 16:44:43.624673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.624642 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" Oct 30 16:44:43.659350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.659320 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zq2w\" (UniqueName: \"kubernetes.io/projected/df9fefa6-cb76-453e-b5c2-37e429f2cd5d-kube-api-access-7zq2w\") pod \"payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9\" (UID: \"df9fefa6-cb76-453e-b5c2-37e429f2cd5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" Oct 30 16:44:43.659502 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.659469 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9fefa6-cb76-453e-b5c2-37e429f2cd5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9\" (UID: \"df9fefa6-cb76-453e-b5c2-37e429f2cd5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" Oct 30 16:44:43.660182 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.660158 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9fefa6-cb76-453e-b5c2-37e429f2cd5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9\" (UID: \"df9fefa6-cb76-453e-b5c2-37e429f2cd5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" Oct 30 16:44:43.685143 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.685107 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zq2w\" (UniqueName: \"kubernetes.io/projected/df9fefa6-cb76-453e-b5c2-37e429f2cd5d-kube-api-access-7zq2w\") pod \"payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9\" (UID: \"df9fefa6-cb76-453e-b5c2-37e429f2cd5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" Oct 30 16:44:43.754180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.754113 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" Oct 30 16:44:43.950367 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:43.950331 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8"] Oct 30 16:44:43.950866 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:44:43.950839 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba88b21_2400_499d_960d_03320e5b9dcd.slice/crio-5ed0b9f70222ed20ef0a09d03cc1863b599010cd00c6d901fb70def3c3a41d7c WatchSource:0}: Error finding container 5ed0b9f70222ed20ef0a09d03cc1863b599010cd00c6d901fb70def3c3a41d7c: Status 404 returned error can't find the container with id 5ed0b9f70222ed20ef0a09d03cc1863b599010cd00c6d901fb70def3c3a41d7c Oct 30 16:44:44.130992 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:44.129040 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9"] Oct 30 16:44:44.132074 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:44:44.132048 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf9fefa6_cb76_453e_b5c2_37e429f2cd5d.slice/crio-10073a912a06332d48f7a42e60ea511089c8544448fb14933547bd8ea4e40ebf WatchSource:0}: Error finding container 10073a912a06332d48f7a42e60ea511089c8544448fb14933547bd8ea4e40ebf: Status 404 returned error can't find the container with id 10073a912a06332d48f7a42e60ea511089c8544448fb14933547bd8ea4e40ebf Oct 30 16:44:44.182620 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:44.182593 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:44.455926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:44.455840 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" Oct 30 16:44:44.459155 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:44.459128 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" Oct 30 16:44:44.736397 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:44.736371 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:44:44.773936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:44.773908 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" event={"ID":"df9fefa6-cb76-453e-b5c2-37e429f2cd5d","Type":"ContainerStarted","Data":"8ab51314ed849e8081bf494857995635f56f437d6630d13597608716dcce90e4"} Oct 30 16:44:44.774336 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:44.773966 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" Oct 30 16:44:44.774336 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:44.773992 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" event={"ID":"df9fefa6-cb76-453e-b5c2-37e429f2cd5d","Type":"ContainerStarted","Data":"10073a912a06332d48f7a42e60ea511089c8544448fb14933547bd8ea4e40ebf"} Oct 30 16:44:44.775701 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:44.775672 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" event={"ID":"4ba88b21-2400-499d-960d-03320e5b9dcd","Type":"ContainerStarted","Data":"a7b8216c50fa63a48a98ecaff3a2e903447f2bc5fee4d56d160b2a3e7d9d7a04"} Oct 30 16:44:44.775865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:44.775736 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" event={"ID":"4ba88b21-2400-499d-960d-03320e5b9dcd","Type":"ContainerStarted","Data":"5ed0b9f70222ed20ef0a09d03cc1863b599010cd00c6d901fb70def3c3a41d7c"} Oct 30 16:44:44.775865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:44.775816 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" Oct 30 16:44:44.887707 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:44.887645 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" podStartSLOduration=1.887628079 podStartE2EDuration="1.887628079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:44.887076018 +0000 UTC m=+1315.229750057" watchObservedRunningTime="2026-10-30 16:44:44.887628079 +0000 UTC m=+1315.230302117" Oct 30 16:44:44.891440 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:44.891402 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" podStartSLOduration=1.891389352 podStartE2EDuration="1.891389352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:44.845861507 +0000 UTC m=+1315.188535547" watchObservedRunningTime="2026-10-30 16:44:44.891389352 +0000 UTC m=+1315.234063422" Oct 30 16:44:46.153842 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:46.153747 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:47.964645 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:47.964613 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:49.784484 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:49.784454 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:49.866769 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:49.866713 4861 generic.go:356] "Generic (PLEG): container finished" podID="e3c4a663-826a-4d68-9208-e16cb76839b5" containerID="6fbed1c15fbce1d4d93b5dde9ac59029a7b9277a09599863fa88ac2558bb4886" exitCode=0 Oct 30 16:44:49.866769 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:49.866748 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" event={"ID":"e3c4a663-826a-4d68-9208-e16cb76839b5","Type":"ContainerDied","Data":"6fbed1c15fbce1d4d93b5dde9ac59029a7b9277a09599863fa88ac2558bb4886"} Oct 30 16:44:49.867190 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:49.867176 4861 scope.go:122] "RemoveContainer" containerID="6fbed1c15fbce1d4d93b5dde9ac59029a7b9277a09599863fa88ac2558bb4886" Oct 30 16:44:49.868979 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:49.868950 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:49.869064 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:49.869026 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:49.869109 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:49.869060 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:50.014627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.014574 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:50.293715 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:44:50.293688 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:44:50.435453 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.435418 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w"] Oct 30 16:44:50.441875 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.441851 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.457535 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.457502 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-dc1ea0-openshift-default-dockercfg-8xqt5\"" Oct 30 16:44:50.481493 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.481435 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w"] Oct 30 16:44:50.533838 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.533800 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:50.622696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.622602 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-workload-socket\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.623074 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.623035 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-data\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.623289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.623270 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-token\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.623591 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.623510 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-podinfo\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.623815 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.623789 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-credential-socket\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.629880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.629801 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7409c5f8-d599-4876-be1a-f5b0a25c0416-istiod-ca-cert\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.630124 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.630105 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-envoy\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.630436 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.630418 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-workload-certs\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.630647 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.630631 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbwdt\" (UniqueName: \"kubernetes.io/projected/7409c5f8-d599-4876-be1a-f5b0a25c0416-kube-api-access-nbwdt\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.751080 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.751036 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-workload-socket\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.752832 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.752674 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-workload-socket\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.753028 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.752789 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-data\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.753601 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.753574 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-token\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.754206 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.753490 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-data\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.754423 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.754386 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-podinfo\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.755173 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.755154 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-credential-socket\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.755737 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.755706 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7409c5f8-d599-4876-be1a-f5b0a25c0416-istiod-ca-cert\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.755918 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.755903 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-envoy\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.759751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.756117 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-workload-certs\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.770216 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.770194 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbwdt\" (UniqueName: \"kubernetes.io/projected/7409c5f8-d599-4876-be1a-f5b0a25c0416-kube-api-access-nbwdt\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.770532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.761029 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7409c5f8-d599-4876-be1a-f5b0a25c0416-istiod-ca-cert\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.770668 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.755653 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-credential-socket\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.770799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.760255 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-workload-certs\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.776350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.776322 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-podinfo\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.787748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.782862 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-envoy\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.791555 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.791505 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-token\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.798636 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.798611 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbwdt\" (UniqueName: \"kubernetes.io/projected/7409c5f8-d599-4876-be1a-f5b0a25c0416-kube-api-access-nbwdt\") pod \"e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:50.875529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:50.875440 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" event={"ID":"e3c4a663-826a-4d68-9208-e16cb76839b5","Type":"ContainerStarted","Data":"8f9ca42bd98cf870601928f2955f9e5320a291767a6df1bd19505c5379cedb8f"} Oct 30 16:44:51.057391 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:51.057364 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:51.192821 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:51.190943 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:51.476748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:51.476065 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:44:51.476748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:51.476493 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 16:44:51.476748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:51.476557 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" podUID="e3c4a663-826a-4d68-9208-e16cb76839b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 16:44:51.545748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:51.539505 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w"] Oct 30 16:44:51.599170 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:51.599125 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:51.599311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:51.599205 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:51.599311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:51.599246 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:51.876969 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:51.876878 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:51.888347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:51.883637 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" event={"ID":"7409c5f8-d599-4876-be1a-f5b0a25c0416","Type":"ContainerStarted","Data":"b18f190561fb3f4d2e9d115f847faa32d71712b3c3781dca6fa74f38b155542e"} Oct 30 16:44:51.888649 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:51.888621 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" event={"ID":"7409c5f8-d599-4876-be1a-f5b0a25c0416","Type":"ContainerStarted","Data":"1d8937d87a83182a483f0659cbd3fc7006227f9d6b0bf3e39df013c413620f9d"} Oct 30 16:44:51.982372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:51.980383 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" podStartSLOduration=1.980280399 podStartE2EDuration="1.980280399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:51.958055274 +0000 UTC m=+1322.300729312" watchObservedRunningTime="2026-10-30 16:44:51.980280399 +0000 UTC m=+1322.322954432" Oct 30 16:44:52.058132 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:52.058097 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:52.059413 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:52.059382 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:44:52.059563 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:52.059441 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" podUID="7409c5f8-d599-4876-be1a-f5b0a25c0416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 16:44:52.472975 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:52.472939 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 16:44:52.473154 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:52.472991 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" podUID="e3c4a663-826a-4d68-9208-e16cb76839b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 16:44:52.887569 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:52.887498 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:53.063649 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:53.063623 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:53.066005 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:53.065983 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c"] Oct 30 16:44:53.099511 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:53.099485 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b"] Oct 30 16:44:53.237525 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:53.237497 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:53.389981 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:53.389948 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:53.472316 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:53.472282 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 16:44:53.472515 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:53.472336 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" podUID="e3c4a663-826a-4d68-9208-e16cb76839b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 16:44:53.987711 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:53.987680 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:53.988868 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:53.988817 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:44:54.138707 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:54.138674 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:54.471984 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:54.471947 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 16:44:54.472163 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:54.472043 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" podUID="e3c4a663-826a-4d68-9208-e16cb76839b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 16:44:54.797267 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:54.797185 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:54.946113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:54.946055 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:55.472546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:55.472508 4861 patch_prober.go:29] interesting pod/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 16:44:55.472990 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:55.472571 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" podUID="e3c4a663-826a-4d68-9208-e16cb76839b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 16:44:55.786014 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:55.785905 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" Oct 30 16:44:55.786395 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:55.786365 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" Oct 30 16:44:56.477334 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.477309 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:44:56.477806 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.477766 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:44:56.478422 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.478402 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:44:56.516351 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.513448 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:56.658715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.658688 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:56.661137 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.661116 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v"] Oct 30 16:44:56.668103 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.668084 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" Oct 30 16:44:56.683609 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.683582 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-dc1ea0-dockercfg-mm842\"" Oct 30 16:44:56.683714 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.683697 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-dc1ea0\"" Oct 30 16:44:56.684876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.684854 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v"] Oct 30 16:44:56.750804 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.747035 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtwzg\" (UniqueName: \"kubernetes.io/projected/74bf6688-428d-49ac-b1a9-b26c65a90696-kube-api-access-gtwzg\") pod \"payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v\" (UID: \"74bf6688-428d-49ac-b1a9-b26c65a90696\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" Oct 30 16:44:56.750804 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.747178 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bf6688-428d-49ac-b1a9-b26c65a90696-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v\" (UID: \"74bf6688-428d-49ac-b1a9-b26c65a90696\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" Oct 30 16:44:56.844519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.826808 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf"] Oct 30 16:44:56.909904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.907311 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:56.946897 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.946851 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtwzg\" (UniqueName: \"kubernetes.io/projected/74bf6688-428d-49ac-b1a9-b26c65a90696-kube-api-access-gtwzg\") pod \"payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v\" (UID: \"74bf6688-428d-49ac-b1a9-b26c65a90696\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" Oct 30 16:44:56.947267 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.947239 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bf6688-428d-49ac-b1a9-b26c65a90696-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v\" (UID: \"74bf6688-428d-49ac-b1a9-b26c65a90696\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" Oct 30 16:44:56.948943 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.948919 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bf6688-428d-49ac-b1a9-b26c65a90696-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v\" (UID: \"74bf6688-428d-49ac-b1a9-b26c65a90696\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" Oct 30 16:44:56.954939 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.954901 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf"] Oct 30 16:44:56.955369 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:56.955337 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" Oct 30 16:44:57.019058 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.016865 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtwzg\" (UniqueName: \"kubernetes.io/projected/74bf6688-428d-49ac-b1a9-b26c65a90696-kube-api-access-gtwzg\") pod \"payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v\" (UID: \"74bf6688-428d-49ac-b1a9-b26c65a90696\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" Oct 30 16:44:57.052884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.052379 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84hsk\" (UniqueName: \"kubernetes.io/projected/8bdc5cff-5719-402a-97a2-c04cbf2d7c23-kube-api-access-84hsk\") pod \"payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf\" (UID: \"8bdc5cff-5719-402a-97a2-c04cbf2d7c23\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" Oct 30 16:44:57.052884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.052451 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdc5cff-5719-402a-97a2-c04cbf2d7c23-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf\" (UID: \"8bdc5cff-5719-402a-97a2-c04cbf2d7c23\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" Oct 30 16:44:57.153836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.153796 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84hsk\" (UniqueName: \"kubernetes.io/projected/8bdc5cff-5719-402a-97a2-c04cbf2d7c23-kube-api-access-84hsk\") pod \"payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf\" (UID: \"8bdc5cff-5719-402a-97a2-c04cbf2d7c23\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" Oct 30 16:44:57.154034 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.153854 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdc5cff-5719-402a-97a2-c04cbf2d7c23-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf\" (UID: \"8bdc5cff-5719-402a-97a2-c04cbf2d7c23\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" Oct 30 16:44:57.154630 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.154604 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdc5cff-5719-402a-97a2-c04cbf2d7c23-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf\" (UID: \"8bdc5cff-5719-402a-97a2-c04cbf2d7c23\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" Oct 30 16:44:57.178098 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.178067 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84hsk\" (UniqueName: \"kubernetes.io/projected/8bdc5cff-5719-402a-97a2-c04cbf2d7c23-kube-api-access-84hsk\") pod \"payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf\" (UID: \"8bdc5cff-5719-402a-97a2-c04cbf2d7c23\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" Oct 30 16:44:57.185767 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.185711 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:57.285143 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.285079 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" Oct 30 16:44:57.359574 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.359541 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" Oct 30 16:44:57.570353 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.570285 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27"] Oct 30 16:44:57.576681 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.576660 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.609511 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.609438 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v"] Oct 30 16:44:57.613064 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.613036 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27"] Oct 30 16:44:57.614761 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.614705 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-64c409-openshift-default-dockercfg-sktr6\"" Oct 30 16:44:57.663529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.663502 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istiod-ca-cert\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.663654 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.663562 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-workload-certs\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.663706 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.663651 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkqsp\" (UniqueName: \"kubernetes.io/projected/68fec9d9-15ed-45c2-95d3-fce8d047a6db-kube-api-access-tkqsp\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.663706 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.663697 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-workload-socket\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.663834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.663798 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-credential-socket\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.663872 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.663854 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-data\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.663929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.663915 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-podinfo\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.664013 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.663938 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-envoy\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.664013 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.663953 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-token\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.671414 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.671393 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:57.766334 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.765844 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-data\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.769894 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.769865 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-podinfo\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.863107 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.863027 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-envoy\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.863339 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.863317 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-token\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.863666 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.863631 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istiod-ca-cert\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.863879 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.863862 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-workload-certs\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.864054 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.864038 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkqsp\" (UniqueName: \"kubernetes.io/projected/68fec9d9-15ed-45c2-95d3-fce8d047a6db-kube-api-access-tkqsp\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.864321 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.864287 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-workload-socket\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.864550 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.864517 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-credential-socket\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.864768 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.813811 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-podinfo\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.864936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.766291 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-data\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.867445 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.865285 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-credential-socket\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.867445 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.786074 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf"] Oct 30 16:44:57.867445 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.865579 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-workload-socket\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.867445 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.866442 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istiod-ca-cert\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.867445 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.867401 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-workload-certs\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.872542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.872519 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-envoy\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.889389 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.889357 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-token\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.890762 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.890713 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkqsp\" (UniqueName: \"kubernetes.io/projected/68fec9d9-15ed-45c2-95d3-fce8d047a6db-kube-api-access-tkqsp\") pod \"e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:57.894796 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:57.894771 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:58.066755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.066574 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" event={"ID":"8bdc5cff-5719-402a-97a2-c04cbf2d7c23","Type":"ContainerStarted","Data":"a3e5377892b6ae58e34e1d12b7c85dba0127866aac869f1687722f356545e939"} Oct 30 16:44:58.066755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.066646 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" event={"ID":"8bdc5cff-5719-402a-97a2-c04cbf2d7c23","Type":"ContainerStarted","Data":"7a2ebe206e6c1cfb16ea33486081ae360c3c9a5409e1276dde59ba1129d1d35f"} Oct 30 16:44:58.084782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.067914 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" Oct 30 16:44:58.084782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.084147 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" event={"ID":"74bf6688-428d-49ac-b1a9-b26c65a90696","Type":"ContainerStarted","Data":"1278828e066f372b9461dd43ad0393ffb8c1e937c012ecddedea1c51cae8d24d"} Oct 30 16:44:58.084782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.084206 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" Oct 30 16:44:58.084782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.084274 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" event={"ID":"74bf6688-428d-49ac-b1a9-b26c65a90696","Type":"ContainerStarted","Data":"1b185f2d29deee819216a4d819a46b174b425defc2daf835f4466e0ac3fa3261"} Oct 30 16:44:58.085147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.084908 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" podUID="ed177420-80fa-483f-a97a-62286e7f20fd" containerName="payload-processing" containerID="cri-o://a32c8622372e6c6f3a50231a352ddbfb399f0f48ec979abf395f9a339c1719d2" gracePeriod=25 Oct 30 16:44:58.086095 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.086073 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:58.100238 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.099776 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" podUID="2fecce1f-7675-40f7-9e2f-a849ceb4bc04" containerName="payload-pre-processing" containerID="cri-o://ede57ab7013a046507ec776946bb89d2cb280e0fd64fa6b0e25cdb7a205d7c60" gracePeriod=25 Oct 30 16:44:58.145887 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.144903 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" podStartSLOduration=2.144886495 podStartE2EDuration="2.144886495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:58.137859839 +0000 UTC m=+1328.480533877" watchObservedRunningTime="2026-10-30 16:44:58.144886495 +0000 UTC m=+1328.487560543" Oct 30 16:44:58.184022 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.183964 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" podStartSLOduration=2.183948279 podStartE2EDuration="2.183948279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:58.181658289 +0000 UTC m=+1328.524332330" watchObservedRunningTime="2026-10-30 16:44:58.183948279 +0000 UTC m=+1328.526622319" Oct 30 16:44:58.228302 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.228274 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27"] Oct 30 16:44:58.234092 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:44:58.234059 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fec9d9_15ed_45c2_95d3_fce8d047a6db.slice/crio-566d1b71b3f88d02df807a4a86a2a4dbc2c6c38b8867d9b779dac40090fac666 WatchSource:0}: Error finding container 566d1b71b3f88d02df807a4a86a2a4dbc2c6c38b8867d9b779dac40090fac666: Status 404 returned error can't find the container with id 566d1b71b3f88d02df807a4a86a2a4dbc2c6c38b8867d9b779dac40090fac666 Oct 30 16:44:58.237583 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.237545 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:58.237675 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.237635 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:58.237716 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.237678 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:44:58.459453 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.459113 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" Oct 30 16:44:58.489023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.488995 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed177420-80fa-483f-a97a-62286e7f20fd-kube-api-access-frptg\" (UniqueName: \"kubernetes.io/projected/ed177420-80fa-483f-a97a-62286e7f20fd-kube-api-access-frptg\") pod \"ed177420-80fa-483f-a97a-62286e7f20fd\" (UID: \"ed177420-80fa-483f-a97a-62286e7f20fd\") " Oct 30 16:44:58.489181 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.489081 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed177420-80fa-483f-a97a-62286e7f20fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed177420-80fa-483f-a97a-62286e7f20fd-plugins-config-volume\") pod \"ed177420-80fa-483f-a97a-62286e7f20fd\" (UID: \"ed177420-80fa-483f-a97a-62286e7f20fd\") " Oct 30 16:44:58.489715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.489688 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed177420-80fa-483f-a97a-62286e7f20fd-plugins-config-volume" pod "ed177420-80fa-483f-a97a-62286e7f20fd" (UID: "ed177420-80fa-483f-a97a-62286e7f20fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:58.493426 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.492033 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed177420-80fa-483f-a97a-62286e7f20fd-kube-api-access-frptg" pod "ed177420-80fa-483f-a97a-62286e7f20fd" (UID: "ed177420-80fa-483f-a97a-62286e7f20fd"). InnerVolumeSpecName "kube-api-access-frptg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:58.505689 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.505662 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" Oct 30 16:44:58.590790 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.590715 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fecce1f-7675-40f7-9e2f-a849ceb4bc04-kube-api-access-56dzn\" (UniqueName: \"kubernetes.io/projected/2fecce1f-7675-40f7-9e2f-a849ceb4bc04-kube-api-access-56dzn\") pod \"2fecce1f-7675-40f7-9e2f-a849ceb4bc04\" (UID: \"2fecce1f-7675-40f7-9e2f-a849ceb4bc04\") " Oct 30 16:44:58.591455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.591424 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fecce1f-7675-40f7-9e2f-a849ceb4bc04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fecce1f-7675-40f7-9e2f-a849ceb4bc04-plugins-config-volume\") pod \"2fecce1f-7675-40f7-9e2f-a849ceb4bc04\" (UID: \"2fecce1f-7675-40f7-9e2f-a849ceb4bc04\") " Oct 30 16:44:58.591917 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.591889 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fecce1f-7675-40f7-9e2f-a849ceb4bc04-plugins-config-volume" pod "2fecce1f-7675-40f7-9e2f-a849ceb4bc04" (UID: "2fecce1f-7675-40f7-9e2f-a849ceb4bc04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:44:58.592104 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.592081 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fecce1f-7675-40f7-9e2f-a849ceb4bc04-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:58.592190 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.592112 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frptg\" (UniqueName: \"kubernetes.io/projected/ed177420-80fa-483f-a97a-62286e7f20fd-kube-api-access-frptg\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:58.592190 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.592127 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed177420-80fa-483f-a97a-62286e7f20fd-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:58.593686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.593660 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fecce1f-7675-40f7-9e2f-a849ceb4bc04-kube-api-access-56dzn" pod "2fecce1f-7675-40f7-9e2f-a849ceb4bc04" (UID: "2fecce1f-7675-40f7-9e2f-a849ceb4bc04"). InnerVolumeSpecName "kube-api-access-56dzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:44:58.693415 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.693347 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56dzn\" (UniqueName: \"kubernetes.io/projected/2fecce1f-7675-40f7-9e2f-a849ceb4bc04-kube-api-access-56dzn\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:44:58.728845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:58.728813 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:59.080667 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.080633 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" event={"ID":"68fec9d9-15ed-45c2-95d3-fce8d047a6db","Type":"ContainerStarted","Data":"1107a716eb0df91b1649fa75a12456e4914cba6a0e27b9c9fe729450f876b864"} Oct 30 16:44:59.080977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.080943 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" event={"ID":"68fec9d9-15ed-45c2-95d3-fce8d047a6db","Type":"ContainerStarted","Data":"566d1b71b3f88d02df807a4a86a2a4dbc2c6c38b8867d9b779dac40090fac666"} Oct 30 16:44:59.082818 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.082780 4861 generic.go:356] "Generic (PLEG): container finished" podID="2fecce1f-7675-40f7-9e2f-a849ceb4bc04" containerID="ede57ab7013a046507ec776946bb89d2cb280e0fd64fa6b0e25cdb7a205d7c60" exitCode=0 Oct 30 16:44:59.082926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.082827 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" Oct 30 16:44:59.082926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.082863 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" event={"ID":"2fecce1f-7675-40f7-9e2f-a849ceb4bc04","Type":"ContainerDied","Data":"ede57ab7013a046507ec776946bb89d2cb280e0fd64fa6b0e25cdb7a205d7c60"} Oct 30 16:44:59.082926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.082912 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b" event={"ID":"2fecce1f-7675-40f7-9e2f-a849ceb4bc04","Type":"ContainerDied","Data":"f65c2fbd753c672980a60ee8a04b35bd60fa7bb18367585b50a8c987e1c9d106"} Oct 30 16:44:59.083077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.082944 4861 scope.go:122] "RemoveContainer" containerID="ede57ab7013a046507ec776946bb89d2cb280e0fd64fa6b0e25cdb7a205d7c60" Oct 30 16:44:59.085365 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.085338 4861 generic.go:356] "Generic (PLEG): container finished" podID="ed177420-80fa-483f-a97a-62286e7f20fd" containerID="a32c8622372e6c6f3a50231a352ddbfb399f0f48ec979abf395f9a339c1719d2" exitCode=0 Oct 30 16:44:59.085464 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.085390 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" Oct 30 16:44:59.085464 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.085434 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" event={"ID":"ed177420-80fa-483f-a97a-62286e7f20fd","Type":"ContainerDied","Data":"a32c8622372e6c6f3a50231a352ddbfb399f0f48ec979abf395f9a339c1719d2"} Oct 30 16:44:59.085564 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.085484 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c" event={"ID":"ed177420-80fa-483f-a97a-62286e7f20fd","Type":"ContainerDied","Data":"32b9a1322d3e1a5329e6b9d807da4a965538c2a56417a44cc5687a4e160ef585"} Oct 30 16:44:59.094822 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.094795 4861 scope.go:122] "RemoveContainer" containerID="ede57ab7013a046507ec776946bb89d2cb280e0fd64fa6b0e25cdb7a205d7c60" Oct 30 16:44:59.095133 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:44:59.095108 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede57ab7013a046507ec776946bb89d2cb280e0fd64fa6b0e25cdb7a205d7c60\": container with ID starting with ede57ab7013a046507ec776946bb89d2cb280e0fd64fa6b0e25cdb7a205d7c60 not found: ID does not exist" containerID="ede57ab7013a046507ec776946bb89d2cb280e0fd64fa6b0e25cdb7a205d7c60" Oct 30 16:44:59.095202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.095142 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede57ab7013a046507ec776946bb89d2cb280e0fd64fa6b0e25cdb7a205d7c60"} err="failed to get container status \"ede57ab7013a046507ec776946bb89d2cb280e0fd64fa6b0e25cdb7a205d7c60\": rpc error: code = NotFound desc = could not find container \"ede57ab7013a046507ec776946bb89d2cb280e0fd64fa6b0e25cdb7a205d7c60\": container with ID starting with ede57ab7013a046507ec776946bb89d2cb280e0fd64fa6b0e25cdb7a205d7c60 not found: ID does not exist" Oct 30 16:44:59.095202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.095159 4861 scope.go:122] "RemoveContainer" containerID="a32c8622372e6c6f3a50231a352ddbfb399f0f48ec979abf395f9a339c1719d2" Oct 30 16:44:59.112539 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.112513 4861 scope.go:122] "RemoveContainer" containerID="a32c8622372e6c6f3a50231a352ddbfb399f0f48ec979abf395f9a339c1719d2" Oct 30 16:44:59.113020 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:44:59.112988 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32c8622372e6c6f3a50231a352ddbfb399f0f48ec979abf395f9a339c1719d2\": container with ID starting with a32c8622372e6c6f3a50231a352ddbfb399f0f48ec979abf395f9a339c1719d2 not found: ID does not exist" containerID="a32c8622372e6c6f3a50231a352ddbfb399f0f48ec979abf395f9a339c1719d2" Oct 30 16:44:59.113176 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.113027 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32c8622372e6c6f3a50231a352ddbfb399f0f48ec979abf395f9a339c1719d2"} err="failed to get container status \"a32c8622372e6c6f3a50231a352ddbfb399f0f48ec979abf395f9a339c1719d2\": rpc error: code = NotFound desc = could not find container \"a32c8622372e6c6f3a50231a352ddbfb399f0f48ec979abf395f9a339c1719d2\": container with ID starting with a32c8622372e6c6f3a50231a352ddbfb399f0f48ec979abf395f9a339c1719d2 not found: ID does not exist" Oct 30 16:44:59.153210 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.153150 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" podStartSLOduration=2.153134919 podStartE2EDuration="2.153134919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:59.139461017 +0000 UTC m=+1329.482135055" watchObservedRunningTime="2026-10-30 16:44:59.153134919 +0000 UTC m=+1329.495808959" Oct 30 16:44:59.179203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.179171 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b"] Oct 30 16:44:59.186602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.186581 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-859cd5575-dpc5b"] Oct 30 16:44:59.225848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.225738 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c"] Oct 30 16:44:59.233930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.233902 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-504442-7b5679bf66-7pv4c"] Oct 30 16:44:59.390261 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.390144 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:59.546588 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.546556 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:59.781268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.781236 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm"] Oct 30 16:44:59.781888 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.781874 4861 cpu_manager.go:414] "containerMap: removing container" podUID="ed177420-80fa-483f-a97a-62286e7f20fd" containerName="payload-processing" Oct 30 16:44:59.781936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.781889 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed177420-80fa-483f-a97a-62286e7f20fd" containerName="payload-processing" Oct 30 16:44:59.781936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.781916 4861 cpu_manager.go:414] "containerMap: removing container" podUID="2fecce1f-7675-40f7-9e2f-a849ceb4bc04" containerName="payload-pre-processing" Oct 30 16:44:59.781936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.781921 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fecce1f-7675-40f7-9e2f-a849ceb4bc04" containerName="payload-pre-processing" Oct 30 16:44:59.782036 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.782022 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed177420-80fa-483f-a97a-62286e7f20fd" containerName="payload-processing" Oct 30 16:44:59.782036 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.782035 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fecce1f-7675-40f7-9e2f-a849ceb4bc04" containerName="payload-pre-processing" Oct 30 16:44:59.787348 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.787327 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.803062 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.803044 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f9970a-openshift-default-dockercfg-xv9vq\"" Oct 30 16:44:59.813639 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.813610 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-token\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.813786 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.813653 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istiod-ca-cert\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.813928 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.813906 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-envoy\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.814016 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.813986 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-data\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.814117 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.814090 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-podinfo\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.814267 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.814249 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-workload-certs\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.814617 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.814592 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-credential-socket\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.814765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.814641 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-workload-socket\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.814765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.814673 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnbwm\" (UniqueName: \"kubernetes.io/projected/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-kube-api-access-bnbwm\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.817380 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.817339 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fecce1f-7675-40f7-9e2f-a849ceb4bc04" path="/var/lib/kubelet/pods/2fecce1f-7675-40f7-9e2f-a849ceb4bc04/volumes" Oct 30 16:44:59.817971 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.817935 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed177420-80fa-483f-a97a-62286e7f20fd" path="/var/lib/kubelet/pods/ed177420-80fa-483f-a97a-62286e7f20fd/volumes" Oct 30 16:44:59.821029 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.821004 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm"] Oct 30 16:44:59.876057 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.876031 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:44:59.914748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.902787 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:44:59.935749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.926716 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-envoy\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.935749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.926823 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-data\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.935749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.926875 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-podinfo\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.935749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.926937 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-workload-certs\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.935749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.927059 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-credential-socket\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.935749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.927097 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-workload-socket\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.935749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.927126 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnbwm\" (UniqueName: \"kubernetes.io/projected/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-kube-api-access-bnbwm\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.935749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.927160 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-token\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.935749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.927183 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istiod-ca-cert\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.950748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.949196 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istiod-ca-cert\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.950748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.949622 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-credential-socket\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.950748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.949918 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-data\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.975675 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.968961 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-workload-certs\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.975675 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.969644 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-workload-socket\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:44:59.975675 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:44:59.973874 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-podinfo\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:45:00.012758 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.010047 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnbwm\" (UniqueName: \"kubernetes.io/projected/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-kube-api-access-bnbwm\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:45:00.030578 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.030547 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-envoy\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:45:00.061312 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.031146 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-token\") pod \"e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:45:00.088750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.088122 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:45:00.113747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.113699 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:45:00.163293 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.160291 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:45:00.163293 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.162429 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:45:00.211620 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.204999 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2"] Oct 30 16:45:00.257896 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.256063 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d"] Oct 30 16:45:00.259013 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.258066 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" Oct 30 16:45:00.266085 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.266061 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2"] Oct 30 16:45:00.266197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.266183 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" Oct 30 16:45:00.266586 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.266569 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d"] Oct 30 16:45:00.281049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.280795 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 16:45:00.347661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.347578 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0b7c46-bcfb-4006-b118-b6bc4051e3e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d\" (UID: \"da0b7c46-bcfb-4006-b118-b6bc4051e3e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" Oct 30 16:45:00.347661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.347635 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb40900-16e1-488a-9718-ff49bc422ffc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2\" (UID: \"0bb40900-16e1-488a-9718-ff49bc422ffc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" Oct 30 16:45:00.347887 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.347814 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cblxd\" (UniqueName: \"kubernetes.io/projected/0bb40900-16e1-488a-9718-ff49bc422ffc-kube-api-access-cblxd\") pod \"maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2\" (UID: \"0bb40900-16e1-488a-9718-ff49bc422ffc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" Oct 30 16:45:00.348045 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.348023 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxz49\" (UniqueName: \"kubernetes.io/projected/da0b7c46-bcfb-4006-b118-b6bc4051e3e9-kube-api-access-kxz49\") pod \"maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d\" (UID: \"da0b7c46-bcfb-4006-b118-b6bc4051e3e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" Oct 30 16:45:00.471747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.461870 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0b7c46-bcfb-4006-b118-b6bc4051e3e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d\" (UID: \"da0b7c46-bcfb-4006-b118-b6bc4051e3e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" Oct 30 16:45:00.471747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.462000 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb40900-16e1-488a-9718-ff49bc422ffc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2\" (UID: \"0bb40900-16e1-488a-9718-ff49bc422ffc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" Oct 30 16:45:00.471747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.462047 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblxd\" (UniqueName: \"kubernetes.io/projected/0bb40900-16e1-488a-9718-ff49bc422ffc-kube-api-access-cblxd\") pod \"maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2\" (UID: \"0bb40900-16e1-488a-9718-ff49bc422ffc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" Oct 30 16:45:00.471747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.462271 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxz49\" (UniqueName: \"kubernetes.io/projected/da0b7c46-bcfb-4006-b118-b6bc4051e3e9-kube-api-access-kxz49\") pod \"maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d\" (UID: \"da0b7c46-bcfb-4006-b118-b6bc4051e3e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" Oct 30 16:45:00.490756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.473629 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb40900-16e1-488a-9718-ff49bc422ffc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2\" (UID: \"0bb40900-16e1-488a-9718-ff49bc422ffc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" Oct 30 16:45:00.490756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.474575 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0b7c46-bcfb-4006-b118-b6bc4051e3e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d\" (UID: \"da0b7c46-bcfb-4006-b118-b6bc4051e3e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" Oct 30 16:45:00.500792 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.500679 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:00.505370 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.505333 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblxd\" (UniqueName: \"kubernetes.io/projected/0bb40900-16e1-488a-9718-ff49bc422ffc-kube-api-access-cblxd\") pod \"maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2\" (UID: \"0bb40900-16e1-488a-9718-ff49bc422ffc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" Oct 30 16:45:00.507646 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.507622 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxz49\" (UniqueName: \"kubernetes.io/projected/da0b7c46-bcfb-4006-b118-b6bc4051e3e9-kube-api-access-kxz49\") pod \"maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d\" (UID: \"da0b7c46-bcfb-4006-b118-b6bc4051e3e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" Oct 30 16:45:00.559595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.559566 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm"] Oct 30 16:45:00.560390 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:45:00.560363 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c27888_3ef3_4b1e_9a85_5d369cf6f92e.slice/crio-deb82e42b125a150a9a0b69857737289a4d726c0b52ec99d5749d2eb1b4c0683 WatchSource:0}: Error finding container deb82e42b125a150a9a0b69857737289a4d726c0b52ec99d5749d2eb1b4c0683: Status 404 returned error can't find the container with id deb82e42b125a150a9a0b69857737289a4d726c0b52ec99d5749d2eb1b4c0683 Oct 30 16:45:00.563103 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.563074 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:45:00.563162 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.563136 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:45:00.563216 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.563166 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:45:00.586457 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.586438 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" Oct 30 16:45:00.592940 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.592924 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" Oct 30 16:45:00.905798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.905705 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2"] Oct 30 16:45:00.912872 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:45:00.912841 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb40900_16e1_488a_9718_ff49bc422ffc.slice/crio-4ef31e53e1287403f76b354c857b2776f59de061534ff4651ce3c81b89e99272 WatchSource:0}: Error finding container 4ef31e53e1287403f76b354c857b2776f59de061534ff4651ce3c81b89e99272: Status 404 returned error can't find the container with id 4ef31e53e1287403f76b354c857b2776f59de061534ff4651ce3c81b89e99272 Oct 30 16:45:00.921629 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:00.921604 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d"] Oct 30 16:45:00.921778 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:45:00.921755 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0b7c46_bcfb_4006_b118_b6bc4051e3e9.slice/crio-445434c5e803a5fffc14a2b86c8ab218e21998b4210b4a2ceccf1c2842fb9fd5 WatchSource:0}: Error finding container 445434c5e803a5fffc14a2b86c8ab218e21998b4210b4a2ceccf1c2842fb9fd5: Status 404 returned error can't find the container with id 445434c5e803a5fffc14a2b86c8ab218e21998b4210b4a2ceccf1c2842fb9fd5 Oct 30 16:45:01.168501 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:01.168397 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" event={"ID":"0bb40900-16e1-488a-9718-ff49bc422ffc","Type":"ContainerStarted","Data":"4ef31e53e1287403f76b354c857b2776f59de061534ff4651ce3c81b89e99272"} Oct 30 16:45:01.169932 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:01.169902 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" event={"ID":"da0b7c46-bcfb-4006-b118-b6bc4051e3e9","Type":"ContainerStarted","Data":"445434c5e803a5fffc14a2b86c8ab218e21998b4210b4a2ceccf1c2842fb9fd5"} Oct 30 16:45:01.171806 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:01.171777 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" event={"ID":"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e","Type":"ContainerStarted","Data":"1e530304e02b6027c5a5de5fdb235e248ddc6011b5b5418b29087724ac1ad666"} Oct 30 16:45:01.171881 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:01.171832 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" event={"ID":"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e","Type":"ContainerStarted","Data":"deb82e42b125a150a9a0b69857737289a4d726c0b52ec99d5749d2eb1b4c0683"} Oct 30 16:45:01.217912 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:01.217870 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" podStartSLOduration=2.217858029 podStartE2EDuration="2.217858029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:01.217593044 +0000 UTC m=+1331.560267087" watchObservedRunningTime="2026-10-30 16:45:01.217858029 +0000 UTC m=+1331.560532067" Oct 30 16:45:01.404463 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:01.404399 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:01.638362 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:01.638292 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:01.793208 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:01.793175 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:02.128329 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:02.128294 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:45:02.133846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:02.133821 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:45:02.180246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:02.180214 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:45:02.181039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:02.181013 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:45:03.539945 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:03.539908 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:03.591450 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:03.591416 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf"] Oct 30 16:45:03.655969 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:03.655939 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf"] Oct 30 16:45:03.656944 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:03.656193 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" Oct 30 16:45:03.656944 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:03.656243 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r"] Oct 30 16:45:03.669120 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:03.669098 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" Oct 30 16:45:03.684274 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:03.684246 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r"] Oct 30 16:45:03.688291 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:03.685345 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-64c409-dockercfg-dgcrq\"" Oct 30 16:45:03.688291 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:03.685454 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-64c409\"" Oct 30 16:45:03.696069 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:03.695579 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:03.908845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:03.908738 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcsk8\" (UniqueName: \"kubernetes.io/projected/b8af0250-b0ec-4590-89c4-3a0ece7ac780-kube-api-access-xcsk8\") pod \"payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf\" (UID: \"b8af0250-b0ec-4590-89c4-3a0ece7ac780\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" Oct 30 16:45:03.909303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:03.909266 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bglw7\" (UniqueName: \"kubernetes.io/projected/97b5bbf9-35cf-48e8-9419-5a1eb03dacfe-kube-api-access-bglw7\") pod \"payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r\" (UID: \"97b5bbf9-35cf-48e8-9419-5a1eb03dacfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" Oct 30 16:45:03.909595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:03.909547 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8af0250-b0ec-4590-89c4-3a0ece7ac780-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf\" (UID: \"b8af0250-b0ec-4590-89c4-3a0ece7ac780\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" Oct 30 16:45:03.909939 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:03.909914 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b5bbf9-35cf-48e8-9419-5a1eb03dacfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r\" (UID: \"97b5bbf9-35cf-48e8-9419-5a1eb03dacfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" Oct 30 16:45:03.980748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:03.980165 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:04.028814 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:04.013656 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcsk8\" (UniqueName: \"kubernetes.io/projected/b8af0250-b0ec-4590-89c4-3a0ece7ac780-kube-api-access-xcsk8\") pod \"payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf\" (UID: \"b8af0250-b0ec-4590-89c4-3a0ece7ac780\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" Oct 30 16:45:04.028814 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:04.013766 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bglw7\" (UniqueName: \"kubernetes.io/projected/97b5bbf9-35cf-48e8-9419-5a1eb03dacfe-kube-api-access-bglw7\") pod \"payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r\" (UID: \"97b5bbf9-35cf-48e8-9419-5a1eb03dacfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" Oct 30 16:45:04.028814 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:04.013840 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8af0250-b0ec-4590-89c4-3a0ece7ac780-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf\" (UID: \"b8af0250-b0ec-4590-89c4-3a0ece7ac780\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" Oct 30 16:45:04.028814 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:04.013964 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b5bbf9-35cf-48e8-9419-5a1eb03dacfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r\" (UID: \"97b5bbf9-35cf-48e8-9419-5a1eb03dacfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" Oct 30 16:45:04.028814 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:04.015070 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b5bbf9-35cf-48e8-9419-5a1eb03dacfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r\" (UID: \"97b5bbf9-35cf-48e8-9419-5a1eb03dacfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" Oct 30 16:45:04.032821 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:04.032789 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8af0250-b0ec-4590-89c4-3a0ece7ac780-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf\" (UID: \"b8af0250-b0ec-4590-89c4-3a0ece7ac780\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" Oct 30 16:45:04.216555 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:04.216478 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcsk8\" (UniqueName: \"kubernetes.io/projected/b8af0250-b0ec-4590-89c4-3a0ece7ac780-kube-api-access-xcsk8\") pod \"payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf\" (UID: \"b8af0250-b0ec-4590-89c4-3a0ece7ac780\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" Oct 30 16:45:04.298584 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:04.298547 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:04.299105 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:04.299077 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" Oct 30 16:45:04.300265 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:04.300242 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bglw7\" (UniqueName: \"kubernetes.io/projected/97b5bbf9-35cf-48e8-9419-5a1eb03dacfe-kube-api-access-bglw7\") pod \"payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r\" (UID: \"97b5bbf9-35cf-48e8-9419-5a1eb03dacfe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" Oct 30 16:45:04.304948 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:04.304912 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" Oct 30 16:45:04.736225 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:04.736191 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf"] Oct 30 16:45:04.752582 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:45:04.752135 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b5bbf9_35cf_48e8_9419_5a1eb03dacfe.slice/crio-88c9c81f739c46c03fcf2f8dad61aa46ab2bac118aa676db4564f9f0f443e141 WatchSource:0}: Error finding container 88c9c81f739c46c03fcf2f8dad61aa46ab2bac118aa676db4564f9f0f443e141: Status 404 returned error can't find the container with id 88c9c81f739c46c03fcf2f8dad61aa46ab2bac118aa676db4564f9f0f443e141 Oct 30 16:45:04.753093 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:04.753064 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r"] Oct 30 16:45:05.332321 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:05.332280 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" event={"ID":"97b5bbf9-35cf-48e8-9419-5a1eb03dacfe","Type":"ContainerStarted","Data":"e17ae1479a0c053430833412616d76f28dd95a29a6454238379ce1ceb9420002"} Oct 30 16:45:05.332519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:05.332350 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" event={"ID":"97b5bbf9-35cf-48e8-9419-5a1eb03dacfe","Type":"ContainerStarted","Data":"88c9c81f739c46c03fcf2f8dad61aa46ab2bac118aa676db4564f9f0f443e141"} Oct 30 16:45:05.332519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:05.332502 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" Oct 30 16:45:05.334104 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:05.334070 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" event={"ID":"b8af0250-b0ec-4590-89c4-3a0ece7ac780","Type":"ContainerStarted","Data":"1ba0aff15e97976aa0cc1480c028dec065ba846b712edb75f51d84caa73efe98"} Oct 30 16:45:05.334222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:05.334116 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" event={"ID":"b8af0250-b0ec-4590-89c4-3a0ece7ac780","Type":"ContainerStarted","Data":"511bea7941f246daab9c920968cbc9725f9754c01403d88ddd8ecf57f241696f"} Oct 30 16:45:05.334222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:05.334150 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" Oct 30 16:45:05.373978 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:05.373931 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" podStartSLOduration=2.373916876 podStartE2EDuration="2.373916876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:05.373579642 +0000 UTC m=+1335.716253680" watchObservedRunningTime="2026-10-30 16:45:05.373916876 +0000 UTC m=+1335.716590947" Oct 30 16:45:05.435376 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:05.435324 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" podStartSLOduration=2.435307881 podStartE2EDuration="2.435307881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:05.430696459 +0000 UTC m=+1335.773370479" watchObservedRunningTime="2026-10-30 16:45:05.435307881 +0000 UTC m=+1335.777981921" Oct 30 16:45:05.777962 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:05.777929 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:05.933425 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:05.933290 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:05.933600 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:05.933465 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v"] Oct 30 16:45:05.938947 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:05.938923 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v"] Oct 30 16:45:05.939099 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:05.939030 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" Oct 30 16:45:05.954882 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:05.954857 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f9970a\"" Oct 30 16:45:05.955198 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:05.955164 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f9970a-dockercfg-gt47l\"" Oct 30 16:45:06.114000 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.105266 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v\" (UID: \"f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" Oct 30 16:45:06.114000 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.105554 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6nmn\" (UniqueName: \"kubernetes.io/projected/f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f-kube-api-access-c6nmn\") pod \"payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v\" (UID: \"f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" Oct 30 16:45:06.276858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.266278 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6nmn\" (UniqueName: \"kubernetes.io/projected/f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f-kube-api-access-c6nmn\") pod \"payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v\" (UID: \"f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" Oct 30 16:45:06.276858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.266353 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v\" (UID: \"f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" Oct 30 16:45:06.276858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.267112 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v\" (UID: \"f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" Oct 30 16:45:06.276858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.276284 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf"] Oct 30 16:45:06.303043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.282965 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" Oct 30 16:45:06.331750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.319787 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf"] Oct 30 16:45:06.331750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.325177 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6nmn\" (UniqueName: \"kubernetes.io/projected/f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f-kube-api-access-c6nmn\") pod \"payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v\" (UID: \"f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" Oct 30 16:45:06.346891 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.346850 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" event={"ID":"0bb40900-16e1-488a-9718-ff49bc422ffc","Type":"ContainerStarted","Data":"64dfd9137ae3da23ba2d942a8f2cc5e31ba7299ec4ac1273138d6135586ff257"} Oct 30 16:45:06.348748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.348680 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" event={"ID":"da0b7c46-bcfb-4006-b118-b6bc4051e3e9","Type":"ContainerStarted","Data":"bb823cc88649255db8f7f31cc2b044fc40eeac2e26b9fecb2314e4472d1fa184"} Oct 30 16:45:06.367678 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.367658 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmmmx\" (UniqueName: \"kubernetes.io/projected/34013d8e-ff30-4d9a-87a4-13b03612b720-kube-api-access-xmmmx\") pod \"payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf\" (UID: \"34013d8e-ff30-4d9a-87a4-13b03612b720\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" Oct 30 16:45:06.367885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.367867 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34013d8e-ff30-4d9a-87a4-13b03612b720-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf\" (UID: \"34013d8e-ff30-4d9a-87a4-13b03612b720\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" Oct 30 16:45:06.391027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.390974 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" podStartSLOduration=1.877499334 podStartE2EDuration="6.390961406s" totalImagesPullingTime="4.513462072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:06.389550051 +0000 UTC m=+1336.732224092" watchObservedRunningTime="2026-10-30 16:45:06.390961406 +0000 UTC m=+1336.733635444" Oct 30 16:45:06.452148 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.452096 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" podStartSLOduration=1.944088323 podStartE2EDuration="6.452078597s" totalImagesPullingTime="4.507990274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:06.450424589 +0000 UTC m=+1336.793098624" watchObservedRunningTime="2026-10-30 16:45:06.452078597 +0000 UTC m=+1336.794752634" Oct 30 16:45:06.469518 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.469489 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmmmx\" (UniqueName: \"kubernetes.io/projected/34013d8e-ff30-4d9a-87a4-13b03612b720-kube-api-access-xmmmx\") pod \"payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf\" (UID: \"34013d8e-ff30-4d9a-87a4-13b03612b720\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" Oct 30 16:45:06.469894 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.469872 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34013d8e-ff30-4d9a-87a4-13b03612b720-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf\" (UID: \"34013d8e-ff30-4d9a-87a4-13b03612b720\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" Oct 30 16:45:06.470629 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.470607 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34013d8e-ff30-4d9a-87a4-13b03612b720-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf\" (UID: \"34013d8e-ff30-4d9a-87a4-13b03612b720\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" Oct 30 16:45:06.493442 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.493422 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmmmx\" (UniqueName: \"kubernetes.io/projected/34013d8e-ff30-4d9a-87a4-13b03612b720-kube-api-access-xmmmx\") pod \"payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf\" (UID: \"34013d8e-ff30-4d9a-87a4-13b03612b720\") " pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" Oct 30 16:45:06.538050 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.537792 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:06.562752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.559100 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" Oct 30 16:45:06.660971 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.660812 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" Oct 30 16:45:06.948154 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:06.948131 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v"] Oct 30 16:45:06.948818 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:45:06.948795 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09deeb3_f361_44de_b4e2_aa7d2a3f2d8f.slice/crio-53ea37becf1bd913b202fc97bbb2b86993e8d396edc57b56a4cd5c53947ad823 WatchSource:0}: Error finding container 53ea37becf1bd913b202fc97bbb2b86993e8d396edc57b56a4cd5c53947ad823: Status 404 returned error can't find the container with id 53ea37becf1bd913b202fc97bbb2b86993e8d396edc57b56a4cd5c53947ad823 Oct 30 16:45:07.031686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:07.031326 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf"] Oct 30 16:45:07.032040 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:45:07.032016 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34013d8e_ff30_4d9a_87a4_13b03612b720.slice/crio-ab2e5f10b9acca0bf82caef7fd647e9534e39147ca15393251e3583c8673f2bb WatchSource:0}: Error finding container ab2e5f10b9acca0bf82caef7fd647e9534e39147ca15393251e3583c8673f2bb: Status 404 returned error can't find the container with id ab2e5f10b9acca0bf82caef7fd647e9534e39147ca15393251e3583c8673f2bb Oct 30 16:45:07.360450 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:07.360411 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" event={"ID":"f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f","Type":"ContainerStarted","Data":"784ceb754bb0f2e820fa1e3a543b9cc2d9ffb2366e0b1bf24966a8aa3481cdc8"} Oct 30 16:45:07.360744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:07.360701 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" event={"ID":"f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f","Type":"ContainerStarted","Data":"53ea37becf1bd913b202fc97bbb2b86993e8d396edc57b56a4cd5c53947ad823"} Oct 30 16:45:07.361638 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:07.361515 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" Oct 30 16:45:07.363306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:07.363268 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" event={"ID":"34013d8e-ff30-4d9a-87a4-13b03612b720","Type":"ContainerStarted","Data":"d962f6b4e40330f6405d1989cdd9672603a90bfb828590247cf44cc115010f1c"} Oct 30 16:45:07.363416 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:07.363339 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" event={"ID":"34013d8e-ff30-4d9a-87a4-13b03612b720","Type":"ContainerStarted","Data":"ab2e5f10b9acca0bf82caef7fd647e9534e39147ca15393251e3583c8673f2bb"} Oct 30 16:45:07.363634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:07.363617 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" Oct 30 16:45:07.366342 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:07.366307 4861 generic.go:356] "Generic (PLEG): container finished" podID="0bb40900-16e1-488a-9718-ff49bc422ffc" containerID="64dfd9137ae3da23ba2d942a8f2cc5e31ba7299ec4ac1273138d6135586ff257" exitCode=60 Oct 30 16:45:07.366503 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:07.366483 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" event={"ID":"0bb40900-16e1-488a-9718-ff49bc422ffc","Type":"ContainerDied","Data":"64dfd9137ae3da23ba2d942a8f2cc5e31ba7299ec4ac1273138d6135586ff257"} Oct 30 16:45:07.367226 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:07.366981 4861 scope.go:122] "RemoveContainer" containerID="64dfd9137ae3da23ba2d942a8f2cc5e31ba7299ec4ac1273138d6135586ff257" Oct 30 16:45:07.369035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:07.369012 4861 generic.go:356] "Generic (PLEG): container finished" podID="da0b7c46-bcfb-4006-b118-b6bc4051e3e9" containerID="bb823cc88649255db8f7f31cc2b044fc40eeac2e26b9fecb2314e4472d1fa184" exitCode=60 Oct 30 16:45:07.369190 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:07.369065 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" event={"ID":"da0b7c46-bcfb-4006-b118-b6bc4051e3e9","Type":"ContainerDied","Data":"bb823cc88649255db8f7f31cc2b044fc40eeac2e26b9fecb2314e4472d1fa184"} Oct 30 16:45:07.370326 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:07.370308 4861 scope.go:122] "RemoveContainer" containerID="bb823cc88649255db8f7f31cc2b044fc40eeac2e26b9fecb2314e4472d1fa184" Oct 30 16:45:07.431593 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:07.431539 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" podStartSLOduration=2.431525294 podStartE2EDuration="2.431525294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:07.427817813 +0000 UTC m=+1337.770491852" watchObservedRunningTime="2026-10-30 16:45:07.431525294 +0000 UTC m=+1337.774199331" Oct 30 16:45:07.606753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:07.586996 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" podStartSLOduration=2.586967123 podStartE2EDuration="2.586967123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:07.579660428 +0000 UTC m=+1337.922334466" watchObservedRunningTime="2026-10-30 16:45:07.586967123 +0000 UTC m=+1337.929641161" Oct 30 16:45:08.379647 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:08.379614 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" event={"ID":"0bb40900-16e1-488a-9718-ff49bc422ffc","Type":"ContainerStarted","Data":"65fbaf894f5e22ad2aab2940da3f244cfca21ec984e4282de5e40005f18643b7"} Oct 30 16:45:08.381570 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:08.381547 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" event={"ID":"da0b7c46-bcfb-4006-b118-b6bc4051e3e9","Type":"ContainerStarted","Data":"e5bee35f8ccabbcde4c8e8289d38d15957aa7b08bc3dd82c8cf89051f87e3003"} Oct 30 16:45:08.853081 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:08.853033 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:08.993092 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:08.993058 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:09.106435 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:09.105077 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" Oct 30 16:45:09.146834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:09.145232 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:09.390199 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:09.390109 4861 generic.go:356] "Generic (PLEG): container finished" podID="0bb40900-16e1-488a-9718-ff49bc422ffc" containerID="65fbaf894f5e22ad2aab2940da3f244cfca21ec984e4282de5e40005f18643b7" exitCode=60 Oct 30 16:45:09.390199 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:09.390181 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" event={"ID":"0bb40900-16e1-488a-9718-ff49bc422ffc","Type":"ContainerDied","Data":"65fbaf894f5e22ad2aab2940da3f244cfca21ec984e4282de5e40005f18643b7"} Oct 30 16:45:09.390743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:09.390251 4861 scope.go:122] "RemoveContainer" containerID="64dfd9137ae3da23ba2d942a8f2cc5e31ba7299ec4ac1273138d6135586ff257" Oct 30 16:45:09.390743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:09.390550 4861 scope.go:122] "RemoveContainer" containerID="65fbaf894f5e22ad2aab2940da3f244cfca21ec984e4282de5e40005f18643b7" Oct 30 16:45:09.390937 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:45:09.390914 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2_odh-ai-gateway-infra(0bb40900-16e1-488a-9718-ff49bc422ffc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" podUID="0bb40900-16e1-488a-9718-ff49bc422ffc" Oct 30 16:45:09.392606 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:09.392580 4861 generic.go:356] "Generic (PLEG): container finished" podID="da0b7c46-bcfb-4006-b118-b6bc4051e3e9" containerID="e5bee35f8ccabbcde4c8e8289d38d15957aa7b08bc3dd82c8cf89051f87e3003" exitCode=60 Oct 30 16:45:09.392702 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:09.392621 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" event={"ID":"da0b7c46-bcfb-4006-b118-b6bc4051e3e9","Type":"ContainerDied","Data":"e5bee35f8ccabbcde4c8e8289d38d15957aa7b08bc3dd82c8cf89051f87e3003"} Oct 30 16:45:09.392915 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:09.392902 4861 scope.go:122] "RemoveContainer" containerID="e5bee35f8ccabbcde4c8e8289d38d15957aa7b08bc3dd82c8cf89051f87e3003" Oct 30 16:45:09.393127 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:45:09.393110 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d_odh-ai-gateway-infra(da0b7c46-bcfb-4006-b118-b6bc4051e3e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" podUID="da0b7c46-bcfb-4006-b118-b6bc4051e3e9" Oct 30 16:45:09.406279 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:09.406254 4861 scope.go:122] "RemoveContainer" containerID="bb823cc88649255db8f7f31cc2b044fc40eeac2e26b9fecb2314e4472d1fa184" Oct 30 16:45:10.163325 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:10.163299 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" Oct 30 16:45:10.401268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:10.401238 4861 scope.go:122] "RemoveContainer" containerID="65fbaf894f5e22ad2aab2940da3f244cfca21ec984e4282de5e40005f18643b7" Oct 30 16:45:10.401766 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:45:10.401595 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2_odh-ai-gateway-infra(0bb40900-16e1-488a-9718-ff49bc422ffc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" podUID="0bb40900-16e1-488a-9718-ff49bc422ffc" Oct 30 16:45:10.403245 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:10.403223 4861 scope.go:122] "RemoveContainer" containerID="e5bee35f8ccabbcde4c8e8289d38d15957aa7b08bc3dd82c8cf89051f87e3003" Oct 30 16:45:10.403486 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:45:10.403468 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d_odh-ai-gateway-infra(da0b7c46-bcfb-4006-b118-b6bc4051e3e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" podUID="da0b7c46-bcfb-4006-b118-b6bc4051e3e9" Oct 30 16:45:11.630130 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:11.630039 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:12.425873 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:12.425834 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:45:12.426039 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:12.425904 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" podUID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:13.429002 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:13.428809 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:13.457396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:13.457357 4861 generic.go:356] "Generic (PLEG): container finished" podID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerID="e1c0aba7496559f921f20473391d3f74e8c5ed5bab855fc9be864c74043cb965" exitCode=0 Oct 30 16:45:13.457627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:13.457588 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" event={"ID":"b7bfc8bc-b483-458e-8580-b32c699fd17b","Type":"ContainerDied","Data":"e1c0aba7496559f921f20473391d3f74e8c5ed5bab855fc9be864c74043cb965"} Oct 30 16:45:13.458335 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:13.458310 4861 scope.go:122] "RemoveContainer" containerID="e1c0aba7496559f921f20473391d3f74e8c5ed5bab855fc9be864c74043cb965" Oct 30 16:45:13.466867 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:13.460841 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:45:13.466867 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:13.460940 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:45:13.466867 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:13.460994 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:45:14.466196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:14.466160 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" event={"ID":"b7bfc8bc-b483-458e-8580-b32c699fd17b","Type":"ContainerStarted","Data":"1b8f9b0624d4be8c381d1cd208566d752a17f871245576d8c365aadfab04fe65"} Oct 30 16:45:14.676110 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:14.676077 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:45:14.676385 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:14.676366 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 16:45:14.676434 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:14.676420 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" podUID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 16:45:15.325827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:15.325482 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:15.675920 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:15.675830 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 16:45:15.675920 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:15.675905 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" podUID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 16:45:16.352885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:16.352856 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" Oct 30 16:45:16.353061 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:16.352938 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" Oct 30 16:45:16.675917 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:16.675833 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 16:45:16.675917 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:16.675887 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" podUID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 16:45:17.275742 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:17.275690 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:17.675671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:17.675582 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 16:45:17.675671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:17.675641 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" podUID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 16:45:18.384363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:18.384337 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" Oct 30 16:45:18.384859 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:18.384399 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" Oct 30 16:45:18.671001 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:18.670899 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:18.676798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:18.676765 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 16:45:18.676949 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:18.676824 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" podUID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 16:45:18.881468 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:18.881335 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:19.337751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.311781 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr"] Oct 30 16:45:19.511427 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.511400 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" Oct 30 16:45:19.545590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.545035 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-504442\"" Oct 30 16:45:19.568695 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.568669 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr"] Oct 30 16:45:19.574372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.574354 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l"] Oct 30 16:45:19.578674 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.578660 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l"] Oct 30 16:45:19.578869 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.578852 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" Oct 30 16:45:19.599641 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.599594 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-504442-dockercfg-m8d2r\"" Oct 30 16:45:19.613938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.613905 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhh4j\" (UniqueName: \"kubernetes.io/projected/73abba4a-c570-436e-a00f-95fcbc80c570-kube-api-access-jhh4j\") pod \"payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l\" (UID: \"73abba4a-c570-436e-a00f-95fcbc80c570\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" Oct 30 16:45:19.614068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.613986 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73abba4a-c570-436e-a00f-95fcbc80c570-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l\" (UID: \"73abba4a-c570-436e-a00f-95fcbc80c570\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" Oct 30 16:45:19.676544 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.676511 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 16:45:19.676716 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.676579 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" podUID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 16:45:19.714958 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.714927 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhh4j\" (UniqueName: \"kubernetes.io/projected/73abba4a-c570-436e-a00f-95fcbc80c570-kube-api-access-jhh4j\") pod \"payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l\" (UID: \"73abba4a-c570-436e-a00f-95fcbc80c570\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" Oct 30 16:45:19.715142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.714975 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b45cs\" (UniqueName: \"kubernetes.io/projected/fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc-kube-api-access-b45cs\") pod \"payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr\" (UID: \"fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" Oct 30 16:45:19.715142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.715041 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73abba4a-c570-436e-a00f-95fcbc80c570-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l\" (UID: \"73abba4a-c570-436e-a00f-95fcbc80c570\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" Oct 30 16:45:19.715242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.715171 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr\" (UID: \"fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" Oct 30 16:45:19.715826 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.715808 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73abba4a-c570-436e-a00f-95fcbc80c570-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l\" (UID: \"73abba4a-c570-436e-a00f-95fcbc80c570\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" Oct 30 16:45:19.740597 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.740571 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhh4j\" (UniqueName: \"kubernetes.io/projected/73abba4a-c570-436e-a00f-95fcbc80c570-kube-api-access-jhh4j\") pod \"payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l\" (UID: \"73abba4a-c570-436e-a00f-95fcbc80c570\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" Oct 30 16:45:19.816232 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.816196 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b45cs\" (UniqueName: \"kubernetes.io/projected/fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc-kube-api-access-b45cs\") pod \"payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr\" (UID: \"fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" Oct 30 16:45:19.816398 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.816324 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr\" (UID: \"fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" Oct 30 16:45:19.817040 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.817016 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr\" (UID: \"fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" Oct 30 16:45:19.841089 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.841053 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b45cs\" (UniqueName: \"kubernetes.io/projected/fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc-kube-api-access-b45cs\") pod \"payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr\" (UID: \"fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" Oct 30 16:45:19.871763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.871659 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" Oct 30 16:45:19.893400 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:19.893362 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" Oct 30 16:45:20.186023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:20.185995 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr"] Oct 30 16:45:20.186571 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:45:20.186547 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef8f9bf_96a1_430f_b3d8_6e9cab2f93dc.slice/crio-3db33cf9f4818875a5b6c317409e89aeb6e82a51023bd8260841958b14e74503 WatchSource:0}: Error finding container 3db33cf9f4818875a5b6c317409e89aeb6e82a51023bd8260841958b14e74503: Status 404 returned error can't find the container with id 3db33cf9f4818875a5b6c317409e89aeb6e82a51023bd8260841958b14e74503 Oct 30 16:45:20.234045 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:20.234023 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l"] Oct 30 16:45:20.235697 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:45:20.235671 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73abba4a_c570_436e_a00f_95fcbc80c570.slice/crio-d68b962f745383e05c5a84e692e6dd816a60855901439c96e2b2d38283f3248f WatchSource:0}: Error finding container d68b962f745383e05c5a84e692e6dd816a60855901439c96e2b2d38283f3248f: Status 404 returned error can't find the container with id d68b962f745383e05c5a84e692e6dd816a60855901439c96e2b2d38283f3248f Oct 30 16:45:20.320854 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:20.319274 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:20.331050 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:45:20.330954 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:45:20.563939 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:20.563898 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" event={"ID":"fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc","Type":"ContainerStarted","Data":"3f89a139dcc134a3653acf0270d7792da4b32d3a5b15084a80a5869a82d121ae"} Oct 30 16:45:20.564432 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:20.563951 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" event={"ID":"fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc","Type":"ContainerStarted","Data":"3db33cf9f4818875a5b6c317409e89aeb6e82a51023bd8260841958b14e74503"} Oct 30 16:45:20.564432 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:20.564103 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" Oct 30 16:45:20.565874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:20.565837 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" event={"ID":"73abba4a-c570-436e-a00f-95fcbc80c570","Type":"ContainerStarted","Data":"9e67d6cef9bcf90783492172c90a45307e9baaaef2a723733b4d7606a442c7a6"} Oct 30 16:45:20.566016 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:20.565883 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" event={"ID":"73abba4a-c570-436e-a00f-95fcbc80c570","Type":"ContainerStarted","Data":"d68b962f745383e05c5a84e692e6dd816a60855901439c96e2b2d38283f3248f"} Oct 30 16:45:20.566016 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:20.565913 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" Oct 30 16:45:20.609577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:20.609521 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" podStartSLOduration=1.609505264 podStartE2EDuration="1.609505264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:20.605360001 +0000 UTC m=+1350.948034039" watchObservedRunningTime="2026-10-30 16:45:20.609505264 +0000 UTC m=+1350.952179302" Oct 30 16:45:20.652321 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:20.652267 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" podStartSLOduration=1.65225354 podStartE2EDuration="1.65225354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:20.648573752 +0000 UTC m=+1350.991247792" watchObservedRunningTime="2026-10-30 16:45:20.65225354 +0000 UTC m=+1350.994927577" Oct 30 16:45:20.679861 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:20.679839 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:45:20.680275 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:20.680251 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:45:20.680941 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:20.680924 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:45:20.803200 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:20.803174 4861 scope.go:122] "RemoveContainer" containerID="e5bee35f8ccabbcde4c8e8289d38d15957aa7b08bc3dd82c8cf89051f87e3003" Oct 30 16:45:21.576145 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:21.576031 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" event={"ID":"da0b7c46-bcfb-4006-b118-b6bc4051e3e9","Type":"ContainerStarted","Data":"79ab7f6d196ce907f0e0a41946a81c1c48c5b1cf869825acb5760d3aeb4bc696"} Oct 30 16:45:21.800682 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:21.800652 4861 scope.go:122] "RemoveContainer" containerID="65fbaf894f5e22ad2aab2940da3f244cfca21ec984e4282de5e40005f18643b7" Oct 30 16:45:22.119592 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:22.119542 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:22.273744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:22.263269 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:22.585592 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:22.585560 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" event={"ID":"0bb40900-16e1-488a-9718-ff49bc422ffc","Type":"ContainerStarted","Data":"7fa88155484c9ebc82e8ff750acc2c740afb2a50ad3e454fe2372167d6e58d19"} Oct 30 16:45:22.587434 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:22.587410 4861 generic.go:356] "Generic (PLEG): container finished" podID="da0b7c46-bcfb-4006-b118-b6bc4051e3e9" containerID="79ab7f6d196ce907f0e0a41946a81c1c48c5b1cf869825acb5760d3aeb4bc696" exitCode=60 Oct 30 16:45:22.587569 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:22.587476 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" event={"ID":"da0b7c46-bcfb-4006-b118-b6bc4051e3e9","Type":"ContainerDied","Data":"79ab7f6d196ce907f0e0a41946a81c1c48c5b1cf869825acb5760d3aeb4bc696"} Oct 30 16:45:22.587569 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:22.587525 4861 scope.go:122] "RemoveContainer" containerID="e5bee35f8ccabbcde4c8e8289d38d15957aa7b08bc3dd82c8cf89051f87e3003" Oct 30 16:45:22.587931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:22.587915 4861 scope.go:122] "RemoveContainer" containerID="79ab7f6d196ce907f0e0a41946a81c1c48c5b1cf869825acb5760d3aeb4bc696" Oct 30 16:45:22.588213 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:45:22.588189 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d_odh-ai-gateway-infra(da0b7c46-bcfb-4006-b118-b6bc4051e3e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" podUID="da0b7c46-bcfb-4006-b118-b6bc4051e3e9" Oct 30 16:45:22.630785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:22.630763 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d"] Oct 30 16:45:23.596082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:23.596047 4861 generic.go:356] "Generic (PLEG): container finished" podID="0bb40900-16e1-488a-9718-ff49bc422ffc" containerID="7fa88155484c9ebc82e8ff750acc2c740afb2a50ad3e454fe2372167d6e58d19" exitCode=60 Oct 30 16:45:23.596564 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:23.596091 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" event={"ID":"0bb40900-16e1-488a-9718-ff49bc422ffc","Type":"ContainerDied","Data":"7fa88155484c9ebc82e8ff750acc2c740afb2a50ad3e454fe2372167d6e58d19"} Oct 30 16:45:23.596564 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:23.596166 4861 scope.go:122] "RemoveContainer" containerID="65fbaf894f5e22ad2aab2940da3f244cfca21ec984e4282de5e40005f18643b7" Oct 30 16:45:23.596564 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:23.596551 4861 scope.go:122] "RemoveContainer" containerID="7fa88155484c9ebc82e8ff750acc2c740afb2a50ad3e454fe2372167d6e58d19" Oct 30 16:45:23.596957 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:45:23.596934 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2_odh-ai-gateway-infra(0bb40900-16e1-488a-9718-ff49bc422ffc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" podUID="0bb40900-16e1-488a-9718-ff49bc422ffc" Oct 30 16:45:23.641684 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:23.641655 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2"] Oct 30 16:45:23.791616 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:23.791590 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" Oct 30 16:45:23.888175 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:23.888102 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0b7c46-bcfb-4006-b118-b6bc4051e3e9-kube-api-access-kxz49\" (UniqueName: \"kubernetes.io/projected/da0b7c46-bcfb-4006-b118-b6bc4051e3e9-kube-api-access-kxz49\") pod \"da0b7c46-bcfb-4006-b118-b6bc4051e3e9\" (UID: \"da0b7c46-bcfb-4006-b118-b6bc4051e3e9\") " Oct 30 16:45:23.888315 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:23.888217 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da0b7c46-bcfb-4006-b118-b6bc4051e3e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0b7c46-bcfb-4006-b118-b6bc4051e3e9-maas-api-service-ca\") pod \"da0b7c46-bcfb-4006-b118-b6bc4051e3e9\" (UID: \"da0b7c46-bcfb-4006-b118-b6bc4051e3e9\") " Oct 30 16:45:23.888585 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:23.888555 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0b7c46-bcfb-4006-b118-b6bc4051e3e9-maas-api-service-ca" pod "da0b7c46-bcfb-4006-b118-b6bc4051e3e9" (UID: "da0b7c46-bcfb-4006-b118-b6bc4051e3e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:23.888751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:23.888674 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0b7c46-bcfb-4006-b118-b6bc4051e3e9-maas-api-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:45:23.890467 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:23.890451 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0b7c46-bcfb-4006-b118-b6bc4051e3e9-kube-api-access-kxz49" pod "da0b7c46-bcfb-4006-b118-b6bc4051e3e9" (UID: "da0b7c46-bcfb-4006-b118-b6bc4051e3e9"). InnerVolumeSpecName "kube-api-access-kxz49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:23.989683 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:23.989649 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxz49\" (UniqueName: \"kubernetes.io/projected/da0b7c46-bcfb-4006-b118-b6bc4051e3e9-kube-api-access-kxz49\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:45:24.234761 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:24.234661 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:24.608407 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:24.608371 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" Oct 30 16:45:24.608826 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:24.608406 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d" event={"ID":"da0b7c46-bcfb-4006-b118-b6bc4051e3e9","Type":"ContainerDied","Data":"445434c5e803a5fffc14a2b86c8ab218e21998b4210b4a2ceccf1c2842fb9fd5"} Oct 30 16:45:24.608826 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:24.608529 4861 scope.go:122] "RemoveContainer" containerID="79ab7f6d196ce907f0e0a41946a81c1c48c5b1cf869825acb5760d3aeb4bc696" Oct 30 16:45:24.692451 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:24.692424 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d"] Oct 30 16:45:24.697933 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:24.697905 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c23a4-29889645-5gd8d"] Oct 30 16:45:24.782496 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:24.782473 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" Oct 30 16:45:24.802238 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:24.802210 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb40900-16e1-488a-9718-ff49bc422ffc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb40900-16e1-488a-9718-ff49bc422ffc-maas-api-service-ca\") pod \"0bb40900-16e1-488a-9718-ff49bc422ffc\" (UID: \"0bb40900-16e1-488a-9718-ff49bc422ffc\") " Oct 30 16:45:24.802349 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:24.802256 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb40900-16e1-488a-9718-ff49bc422ffc-kube-api-access-cblxd\" (UniqueName: \"kubernetes.io/projected/0bb40900-16e1-488a-9718-ff49bc422ffc-kube-api-access-cblxd\") pod \"0bb40900-16e1-488a-9718-ff49bc422ffc\" (UID: \"0bb40900-16e1-488a-9718-ff49bc422ffc\") " Oct 30 16:45:24.802567 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:24.802544 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb40900-16e1-488a-9718-ff49bc422ffc-maas-api-service-ca" pod "0bb40900-16e1-488a-9718-ff49bc422ffc" (UID: "0bb40900-16e1-488a-9718-ff49bc422ffc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:24.802874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:24.802856 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb40900-16e1-488a-9718-ff49bc422ffc-maas-api-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:45:24.804488 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:24.804464 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb40900-16e1-488a-9718-ff49bc422ffc-kube-api-access-cblxd" pod "0bb40900-16e1-488a-9718-ff49bc422ffc" (UID: "0bb40900-16e1-488a-9718-ff49bc422ffc"). InnerVolumeSpecName "kube-api-access-cblxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:24.903672 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:24.903593 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cblxd\" (UniqueName: \"kubernetes.io/projected/0bb40900-16e1-488a-9718-ff49bc422ffc-kube-api-access-cblxd\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:45:25.616658 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:25.616616 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" Oct 30 16:45:25.616658 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:25.616632 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2" event={"ID":"0bb40900-16e1-488a-9718-ff49bc422ffc","Type":"ContainerDied","Data":"4ef31e53e1287403f76b354c857b2776f59de061534ff4651ce3c81b89e99272"} Oct 30 16:45:25.617203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:25.616706 4861 scope.go:122] "RemoveContainer" containerID="7fa88155484c9ebc82e8ff750acc2c740afb2a50ad3e454fe2372167d6e58d19" Oct 30 16:45:25.672285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:25.672261 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2"] Oct 30 16:45:25.684503 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:25.684483 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-843ab3-29889645-bv7j2"] Oct 30 16:45:25.807967 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:25.807943 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb40900-16e1-488a-9718-ff49bc422ffc" path="/var/lib/kubelet/pods/0bb40900-16e1-488a-9718-ff49bc422ffc/volumes" Oct 30 16:45:25.808367 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:25.808354 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0b7c46-bcfb-4006-b118-b6bc4051e3e9" path="/var/lib/kubelet/pods/da0b7c46-bcfb-4006-b118-b6bc4051e3e9/volumes" Oct 30 16:45:26.670748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:26.670453 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:29.000925 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:29.000835 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:30.773269 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.773242 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj"] Oct 30 16:45:30.774013 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.773997 4861 cpu_manager.go:414] "containerMap: removing container" podUID="0bb40900-16e1-488a-9718-ff49bc422ffc" containerName="cleanup" Oct 30 16:45:30.774079 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.774014 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb40900-16e1-488a-9718-ff49bc422ffc" containerName="cleanup" Oct 30 16:45:30.774079 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.774029 4861 cpu_manager.go:414] "containerMap: removing container" podUID="da0b7c46-bcfb-4006-b118-b6bc4051e3e9" containerName="cleanup" Oct 30 16:45:30.774079 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.774034 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0b7c46-bcfb-4006-b118-b6bc4051e3e9" containerName="cleanup" Oct 30 16:45:30.774079 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.774057 4861 cpu_manager.go:414] "containerMap: removing container" podUID="da0b7c46-bcfb-4006-b118-b6bc4051e3e9" containerName="cleanup" Oct 30 16:45:30.774079 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.774061 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0b7c46-bcfb-4006-b118-b6bc4051e3e9" containerName="cleanup" Oct 30 16:45:30.774237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.774086 4861 cpu_manager.go:414] "containerMap: removing container" podUID="0bb40900-16e1-488a-9718-ff49bc422ffc" containerName="cleanup" Oct 30 16:45:30.774237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.774090 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb40900-16e1-488a-9718-ff49bc422ffc" containerName="cleanup" Oct 30 16:45:30.774237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.774189 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0b7c46-bcfb-4006-b118-b6bc4051e3e9" containerName="cleanup" Oct 30 16:45:30.774237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.774199 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb40900-16e1-488a-9718-ff49bc422ffc" containerName="cleanup" Oct 30 16:45:30.774237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.774209 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb40900-16e1-488a-9718-ff49bc422ffc" containerName="cleanup" Oct 30 16:45:30.774237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.774216 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0b7c46-bcfb-4006-b118-b6bc4051e3e9" containerName="cleanup" Oct 30 16:45:30.779290 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.779271 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:45:30.800755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.800702 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj"] Oct 30 16:45:30.801438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.801414 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-64c409-serving-cert\"" Oct 30 16:45:30.881477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.881443 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd472a2-210b-425b-80c3-0fa8ee391ca8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj\" (UID: \"acd472a2-210b-425b-80c3-0fa8ee391ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:45:30.881633 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.881485 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acd472a2-210b-425b-80c3-0fa8ee391ca8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj\" (UID: \"acd472a2-210b-425b-80c3-0fa8ee391ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:45:30.881673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.881635 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx6q4\" (UniqueName: \"kubernetes.io/projected/acd472a2-210b-425b-80c3-0fa8ee391ca8-kube-api-access-nx6q4\") pod \"maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj\" (UID: \"acd472a2-210b-425b-80c3-0fa8ee391ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:45:30.881711 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.881694 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acd472a2-210b-425b-80c3-0fa8ee391ca8-metrics-certs\") pod \"maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj\" (UID: \"acd472a2-210b-425b-80c3-0fa8ee391ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:45:30.892923 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.892882 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:30.985747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.983124 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd472a2-210b-425b-80c3-0fa8ee391ca8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj\" (UID: \"acd472a2-210b-425b-80c3-0fa8ee391ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:45:30.985747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.983186 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acd472a2-210b-425b-80c3-0fa8ee391ca8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj\" (UID: \"acd472a2-210b-425b-80c3-0fa8ee391ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:45:30.985747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.983309 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx6q4\" (UniqueName: \"kubernetes.io/projected/acd472a2-210b-425b-80c3-0fa8ee391ca8-kube-api-access-nx6q4\") pod \"maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj\" (UID: \"acd472a2-210b-425b-80c3-0fa8ee391ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:45:30.985747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:30.983389 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acd472a2-210b-425b-80c3-0fa8ee391ca8-metrics-certs\") pod \"maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj\" (UID: \"acd472a2-210b-425b-80c3-0fa8ee391ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:45:31.006894 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:31.006854 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd472a2-210b-425b-80c3-0fa8ee391ca8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj\" (UID: \"acd472a2-210b-425b-80c3-0fa8ee391ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:45:31.009793 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:31.009413 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acd472a2-210b-425b-80c3-0fa8ee391ca8-metrics-certs\") pod \"maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj\" (UID: \"acd472a2-210b-425b-80c3-0fa8ee391ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:45:31.025025 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:31.024493 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acd472a2-210b-425b-80c3-0fa8ee391ca8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj\" (UID: \"acd472a2-210b-425b-80c3-0fa8ee391ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:45:31.048401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:31.046768 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx6q4\" (UniqueName: \"kubernetes.io/projected/acd472a2-210b-425b-80c3-0fa8ee391ca8-kube-api-access-nx6q4\") pod \"maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj\" (UID: \"acd472a2-210b-425b-80c3-0fa8ee391ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:45:31.050203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:31.050112 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:31.095491 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:31.095455 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:45:31.411998 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:31.411973 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj"] Oct 30 16:45:31.412268 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:45:31.412249 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd472a2_210b_425b_80c3_0fa8ee391ca8.slice/crio-f4cd385cc5d245bb2be1ccd7f310fd1af5f07ecf643003cd68767bbde16bb1bd WatchSource:0}: Error finding container f4cd385cc5d245bb2be1ccd7f310fd1af5f07ecf643003cd68767bbde16bb1bd: Status 404 returned error can't find the container with id f4cd385cc5d245bb2be1ccd7f310fd1af5f07ecf643003cd68767bbde16bb1bd Oct 30 16:45:31.578915 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:31.578882 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" Oct 30 16:45:31.579066 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:31.578957 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" Oct 30 16:45:31.665781 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:31.665649 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" event={"ID":"acd472a2-210b-425b-80c3-0fa8ee391ca8","Type":"ContainerStarted","Data":"f4cd385cc5d245bb2be1ccd7f310fd1af5f07ecf643003cd68767bbde16bb1bd"} Oct 30 16:45:32.675797 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:32.675712 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" event={"ID":"acd472a2-210b-425b-80c3-0fa8ee391ca8","Type":"ContainerStarted","Data":"4d53cfecaf9361f7758641fe953f8a067c3b0f615e50e78150961925c7925233"} Oct 30 16:45:32.676282 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:32.675843 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:45:32.722784 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:32.722713 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" podStartSLOduration=2.426506731 podStartE2EDuration="2.722700595s" totalImagesPullingTime="296.193864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:32.719332357 +0000 UTC m=+1363.062006396" watchObservedRunningTime="2026-10-30 16:45:32.722700595 +0000 UTC m=+1363.065374685" Oct 30 16:45:32.885874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:32.885842 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:34.728942 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:34.728909 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:34.873195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:34.873159 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:35.077160 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:35.077075 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:35.222573 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:35.222542 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:35.700863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:35.700830 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:36.253304 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:36.253270 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:36.864372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:36.864326 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:38.424617 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:38.424580 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:38.674863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:38.662285 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:38.692179 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:38.692146 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:45:38.989491 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:38.989424 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:39.136918 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.136876 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:39.153664 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.149472 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn"] Oct 30 16:45:39.153664 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.150884 4861 cpu_manager.go:414] "containerMap: removing container" podUID="da0b7c46-bcfb-4006-b118-b6bc4051e3e9" containerName="cleanup" Oct 30 16:45:39.153664 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.150906 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0b7c46-bcfb-4006-b118-b6bc4051e3e9" containerName="cleanup" Oct 30 16:45:39.153664 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.150960 4861 cpu_manager.go:414] "containerMap: removing container" podUID="0bb40900-16e1-488a-9718-ff49bc422ffc" containerName="cleanup" Oct 30 16:45:39.153664 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.150968 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb40900-16e1-488a-9718-ff49bc422ffc" containerName="cleanup" Oct 30 16:45:39.153664 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.151168 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0b7c46-bcfb-4006-b118-b6bc4051e3e9" containerName="cleanup" Oct 30 16:45:39.153664 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.151186 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb40900-16e1-488a-9718-ff49bc422ffc" containerName="cleanup" Oct 30 16:45:39.155677 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.155653 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" Oct 30 16:45:39.172899 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.172665 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn"] Oct 30 16:45:39.175394 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.175365 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-4a4c68\"" Oct 30 16:45:39.211776 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.210285 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2d6d0c-3ced-4d69-bfc5-732bcf199453-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn\" (UID: \"af2d6d0c-3ced-4d69-bfc5-732bcf199453\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" Oct 30 16:45:39.211776 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.210371 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5phhb\" (UniqueName: \"kubernetes.io/projected/af2d6d0c-3ced-4d69-bfc5-732bcf199453-kube-api-access-5phhb\") pod \"payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn\" (UID: \"af2d6d0c-3ced-4d69-bfc5-732bcf199453\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" Oct 30 16:45:39.238186 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.238151 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-4a4c68-dockercfg-gq6cr\"" Oct 30 16:45:39.311492 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.311404 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2d6d0c-3ced-4d69-bfc5-732bcf199453-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn\" (UID: \"af2d6d0c-3ced-4d69-bfc5-732bcf199453\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" Oct 30 16:45:39.311492 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.311456 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5phhb\" (UniqueName: \"kubernetes.io/projected/af2d6d0c-3ced-4d69-bfc5-732bcf199453-kube-api-access-5phhb\") pod \"payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn\" (UID: \"af2d6d0c-3ced-4d69-bfc5-732bcf199453\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" Oct 30 16:45:39.312147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.312127 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2d6d0c-3ced-4d69-bfc5-732bcf199453-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn\" (UID: \"af2d6d0c-3ced-4d69-bfc5-732bcf199453\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" Oct 30 16:45:39.335579 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.335550 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5phhb\" (UniqueName: \"kubernetes.io/projected/af2d6d0c-3ced-4d69-bfc5-732bcf199453-kube-api-access-5phhb\") pod \"payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn\" (UID: \"af2d6d0c-3ced-4d69-bfc5-732bcf199453\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" Oct 30 16:45:39.485580 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.485553 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" Oct 30 16:45:39.793846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.793817 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn"] Oct 30 16:45:39.794227 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:45:39.794204 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2d6d0c_3ced_4d69_bfc5_732bcf199453.slice/crio-98fecab976d246156030dbbbe8d11ad5a00270a06d0b132ec1338504d6ae3ce3 WatchSource:0}: Error finding container 98fecab976d246156030dbbbe8d11ad5a00270a06d0b132ec1338504d6ae3ce3: Status 404 returned error can't find the container with id 98fecab976d246156030dbbbe8d11ad5a00270a06d0b132ec1338504d6ae3ce3 Oct 30 16:45:39.902668 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.902637 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" event={"ID":"af2d6d0c-3ced-4d69-bfc5-732bcf199453","Type":"ContainerStarted","Data":"70637b22a082da0f38c78b2fcbac691fb7493796846865b8faabdf4a96e381a5"} Oct 30 16:45:39.902825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.902705 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" Oct 30 16:45:39.902825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.902764 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" event={"ID":"af2d6d0c-3ced-4d69-bfc5-732bcf199453","Type":"ContainerStarted","Data":"98fecab976d246156030dbbbe8d11ad5a00270a06d0b132ec1338504d6ae3ce3"} Oct 30 16:45:39.948450 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:39.948399 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" podStartSLOduration=0.948383847 podStartE2EDuration="948.383847ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:39.945656936 +0000 UTC m=+1370.288330974" watchObservedRunningTime="2026-10-30 16:45:39.948383847 +0000 UTC m=+1370.291057894" Oct 30 16:45:40.133435 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:40.133361 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:40.817387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:40.817358 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:40.971298 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:40.971271 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:41.175900 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:41.161452 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22"] Oct 30 16:45:41.223746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:41.221258 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22"] Oct 30 16:45:41.223746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:41.221641 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" Oct 30 16:45:41.280057 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:41.280023 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-06ac18\"" Oct 30 16:45:41.281477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:41.280788 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-06ac18-dockercfg-zbnrz\"" Oct 30 16:45:41.392365 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:41.392328 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-795wt\" (UniqueName: \"kubernetes.io/projected/6ecf9d2f-4389-46e8-801d-970f54fe8a14-kube-api-access-795wt\") pod \"payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22\" (UID: \"6ecf9d2f-4389-46e8-801d-970f54fe8a14\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" Oct 30 16:45:41.392535 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:41.392482 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecf9d2f-4389-46e8-801d-970f54fe8a14-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22\" (UID: \"6ecf9d2f-4389-46e8-801d-970f54fe8a14\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" Oct 30 16:45:41.494240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:41.494209 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecf9d2f-4389-46e8-801d-970f54fe8a14-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22\" (UID: \"6ecf9d2f-4389-46e8-801d-970f54fe8a14\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" Oct 30 16:45:41.494428 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:41.494410 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-795wt\" (UniqueName: \"kubernetes.io/projected/6ecf9d2f-4389-46e8-801d-970f54fe8a14-kube-api-access-795wt\") pod \"payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22\" (UID: \"6ecf9d2f-4389-46e8-801d-970f54fe8a14\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" Oct 30 16:45:41.494951 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:41.494930 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecf9d2f-4389-46e8-801d-970f54fe8a14-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22\" (UID: \"6ecf9d2f-4389-46e8-801d-970f54fe8a14\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" Oct 30 16:45:41.518215 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:41.518192 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-795wt\" (UniqueName: \"kubernetes.io/projected/6ecf9d2f-4389-46e8-801d-970f54fe8a14-kube-api-access-795wt\") pod \"payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22\" (UID: \"6ecf9d2f-4389-46e8-801d-970f54fe8a14\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" Oct 30 16:45:41.620125 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:41.620092 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" Oct 30 16:45:41.932215 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:41.932192 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22"] Oct 30 16:45:41.933074 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:45:41.933051 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecf9d2f_4389_46e8_801d_970f54fe8a14.slice/crio-cf80523a03d91abda60352c688160c2cdd35182d9451df54e0befa94fff2847c WatchSource:0}: Error finding container cf80523a03d91abda60352c688160c2cdd35182d9451df54e0befa94fff2847c: Status 404 returned error can't find the container with id cf80523a03d91abda60352c688160c2cdd35182d9451df54e0befa94fff2847c Oct 30 16:45:42.136426 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:42.136391 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:42.924120 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:42.924085 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" event={"ID":"6ecf9d2f-4389-46e8-801d-970f54fe8a14","Type":"ContainerStarted","Data":"2669575e75923b99d4d2a4012a18a9298955b00e0dcbdd316527f7e2a3507766"} Oct 30 16:45:42.924286 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:42.924149 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" event={"ID":"6ecf9d2f-4389-46e8-801d-970f54fe8a14","Type":"ContainerStarted","Data":"cf80523a03d91abda60352c688160c2cdd35182d9451df54e0befa94fff2847c"} Oct 30 16:45:42.924286 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:42.924183 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" Oct 30 16:45:42.968587 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:42.968540 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" podStartSLOduration=1.96852614 podStartE2EDuration="1.96852614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:42.967705084 +0000 UTC m=+1373.310379134" watchObservedRunningTime="2026-10-30 16:45:42.96852614 +0000 UTC m=+1373.311200178" Oct 30 16:45:43.415584 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:43.415550 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:44.505466 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:44.505437 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:46.373964 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:46.373932 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:48.195984 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:48.195947 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:50.025927 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:50.025894 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:50.290638 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:45:50.290552 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:45:50.362216 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:50.362186 4861 scope.go:122] "RemoveContainer" containerID="95ed41af8e4ce15bd5650de6970a3f7d43bb8e5343e1042fe7adece440da3045" Oct 30 16:45:50.912787 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:50.912754 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" Oct 30 16:45:52.379746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:52.379461 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:53.934484 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:53.934453 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" Oct 30 16:45:54.064186 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:54.064128 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:55.859130 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:55.859096 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:45:59.115626 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:45:59.115586 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:00.655876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:00.655772 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf"] Oct 30 16:46:00.661246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:00.661206 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v"] Oct 30 16:46:00.848025 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:00.847991 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:01.016640 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:01.016546 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:02.123367 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:02.123274 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:04.006751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:04.001178 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:05.675776 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:05.675523 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" podUID="74bf6688-428d-49ac-b1a9-b26c65a90696" containerName="payload-pre-processing" containerID="cri-o://1278828e066f372b9461dd43ad0393ffb8c1e937c012ecddedea1c51cae8d24d" gracePeriod=25 Oct 30 16:46:05.675776 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:05.675633 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" podUID="8bdc5cff-5719-402a-97a2-c04cbf2d7c23" containerName="payload-processing" containerID="cri-o://a3e5377892b6ae58e34e1d12b7c85dba0127866aac869f1687722f356545e939" gracePeriod=25 Oct 30 16:46:05.907515 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:05.907355 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:06.103134 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.103102 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" Oct 30 16:46:06.161277 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.161209 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" Oct 30 16:46:06.182651 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.182611 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bdc5cff-5719-402a-97a2-c04cbf2d7c23-kube-api-access-84hsk\" (UniqueName: \"kubernetes.io/projected/8bdc5cff-5719-402a-97a2-c04cbf2d7c23-kube-api-access-84hsk\") pod \"8bdc5cff-5719-402a-97a2-c04cbf2d7c23\" (UID: \"8bdc5cff-5719-402a-97a2-c04cbf2d7c23\") " Oct 30 16:46:06.183066 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.182713 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bdc5cff-5719-402a-97a2-c04cbf2d7c23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdc5cff-5719-402a-97a2-c04cbf2d7c23-plugins-config-volume\") pod \"8bdc5cff-5719-402a-97a2-c04cbf2d7c23\" (UID: \"8bdc5cff-5719-402a-97a2-c04cbf2d7c23\") " Oct 30 16:46:06.183539 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.183508 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdc5cff-5719-402a-97a2-c04cbf2d7c23-plugins-config-volume" pod "8bdc5cff-5719-402a-97a2-c04cbf2d7c23" (UID: "8bdc5cff-5719-402a-97a2-c04cbf2d7c23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:06.186302 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.186263 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdc5cff-5719-402a-97a2-c04cbf2d7c23-kube-api-access-84hsk" pod "8bdc5cff-5719-402a-97a2-c04cbf2d7c23" (UID: "8bdc5cff-5719-402a-97a2-c04cbf2d7c23"). InnerVolumeSpecName "kube-api-access-84hsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:06.284795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.284689 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74bf6688-428d-49ac-b1a9-b26c65a90696-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bf6688-428d-49ac-b1a9-b26c65a90696-plugins-config-volume\") pod \"74bf6688-428d-49ac-b1a9-b26c65a90696\" (UID: \"74bf6688-428d-49ac-b1a9-b26c65a90696\") " Oct 30 16:46:06.284985 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.284955 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74bf6688-428d-49ac-b1a9-b26c65a90696-kube-api-access-gtwzg\" (UniqueName: \"kubernetes.io/projected/74bf6688-428d-49ac-b1a9-b26c65a90696-kube-api-access-gtwzg\") pod \"74bf6688-428d-49ac-b1a9-b26c65a90696\" (UID: \"74bf6688-428d-49ac-b1a9-b26c65a90696\") " Oct 30 16:46:06.285154 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.285129 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74bf6688-428d-49ac-b1a9-b26c65a90696-plugins-config-volume" pod "74bf6688-428d-49ac-b1a9-b26c65a90696" (UID: "74bf6688-428d-49ac-b1a9-b26c65a90696"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:06.285703 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.285677 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bdc5cff-5719-402a-97a2-c04cbf2d7c23-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:46:06.285888 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.285707 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74bf6688-428d-49ac-b1a9-b26c65a90696-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:46:06.285888 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.285744 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84hsk\" (UniqueName: \"kubernetes.io/projected/8bdc5cff-5719-402a-97a2-c04cbf2d7c23-kube-api-access-84hsk\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:46:06.287381 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.287358 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74bf6688-428d-49ac-b1a9-b26c65a90696-kube-api-access-gtwzg" pod "74bf6688-428d-49ac-b1a9-b26c65a90696" (UID: "74bf6688-428d-49ac-b1a9-b26c65a90696"). InnerVolumeSpecName "kube-api-access-gtwzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:06.315774 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.315742 4861 generic.go:356] "Generic (PLEG): container finished" podID="8bdc5cff-5719-402a-97a2-c04cbf2d7c23" containerID="a3e5377892b6ae58e34e1d12b7c85dba0127866aac869f1687722f356545e939" exitCode=0 Oct 30 16:46:06.315946 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.315828 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" Oct 30 16:46:06.315946 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.315860 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" event={"ID":"8bdc5cff-5719-402a-97a2-c04cbf2d7c23","Type":"ContainerDied","Data":"a3e5377892b6ae58e34e1d12b7c85dba0127866aac869f1687722f356545e939"} Oct 30 16:46:06.315946 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.315914 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf" event={"ID":"8bdc5cff-5719-402a-97a2-c04cbf2d7c23","Type":"ContainerDied","Data":"7a2ebe206e6c1cfb16ea33486081ae360c3c9a5409e1276dde59ba1129d1d35f"} Oct 30 16:46:06.316096 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.315941 4861 scope.go:122] "RemoveContainer" containerID="a3e5377892b6ae58e34e1d12b7c85dba0127866aac869f1687722f356545e939" Oct 30 16:46:06.318113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.317975 4861 generic.go:356] "Generic (PLEG): container finished" podID="74bf6688-428d-49ac-b1a9-b26c65a90696" containerID="1278828e066f372b9461dd43ad0393ffb8c1e937c012ecddedea1c51cae8d24d" exitCode=0 Oct 30 16:46:06.318113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.318034 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" Oct 30 16:46:06.318113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.318039 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" event={"ID":"74bf6688-428d-49ac-b1a9-b26c65a90696","Type":"ContainerDied","Data":"1278828e066f372b9461dd43ad0393ffb8c1e937c012ecddedea1c51cae8d24d"} Oct 30 16:46:06.318113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.318079 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v" event={"ID":"74bf6688-428d-49ac-b1a9-b26c65a90696","Type":"ContainerDied","Data":"1b185f2d29deee819216a4d819a46b174b425defc2daf835f4466e0ac3fa3261"} Oct 30 16:46:06.330199 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.330183 4861 scope.go:122] "RemoveContainer" containerID="a3e5377892b6ae58e34e1d12b7c85dba0127866aac869f1687722f356545e939" Oct 30 16:46:06.330554 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:46:06.330530 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e5377892b6ae58e34e1d12b7c85dba0127866aac869f1687722f356545e939\": container with ID starting with a3e5377892b6ae58e34e1d12b7c85dba0127866aac869f1687722f356545e939 not found: ID does not exist" containerID="a3e5377892b6ae58e34e1d12b7c85dba0127866aac869f1687722f356545e939" Oct 30 16:46:06.330648 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.330567 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e5377892b6ae58e34e1d12b7c85dba0127866aac869f1687722f356545e939"} err="failed to get container status \"a3e5377892b6ae58e34e1d12b7c85dba0127866aac869f1687722f356545e939\": rpc error: code = NotFound desc = could not find container \"a3e5377892b6ae58e34e1d12b7c85dba0127866aac869f1687722f356545e939\": container with ID starting with a3e5377892b6ae58e34e1d12b7c85dba0127866aac869f1687722f356545e939 not found: ID does not exist" Oct 30 16:46:06.330648 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.330584 4861 scope.go:122] "RemoveContainer" containerID="1278828e066f372b9461dd43ad0393ffb8c1e937c012ecddedea1c51cae8d24d" Oct 30 16:46:06.341882 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.341857 4861 scope.go:122] "RemoveContainer" containerID="1278828e066f372b9461dd43ad0393ffb8c1e937c012ecddedea1c51cae8d24d" Oct 30 16:46:06.342213 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:46:06.342188 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1278828e066f372b9461dd43ad0393ffb8c1e937c012ecddedea1c51cae8d24d\": container with ID starting with 1278828e066f372b9461dd43ad0393ffb8c1e937c012ecddedea1c51cae8d24d not found: ID does not exist" containerID="1278828e066f372b9461dd43ad0393ffb8c1e937c012ecddedea1c51cae8d24d" Oct 30 16:46:06.342300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.342220 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1278828e066f372b9461dd43ad0393ffb8c1e937c012ecddedea1c51cae8d24d"} err="failed to get container status \"1278828e066f372b9461dd43ad0393ffb8c1e937c012ecddedea1c51cae8d24d\": rpc error: code = NotFound desc = could not find container \"1278828e066f372b9461dd43ad0393ffb8c1e937c012ecddedea1c51cae8d24d\": container with ID starting with 1278828e066f372b9461dd43ad0393ffb8c1e937c012ecddedea1c51cae8d24d not found: ID does not exist" Oct 30 16:46:06.377138 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.377101 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v"] Oct 30 16:46:06.383588 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.383557 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-699bfbdc79-l2d5v"] Oct 30 16:46:06.387196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.387170 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtwzg\" (UniqueName: \"kubernetes.io/projected/74bf6688-428d-49ac-b1a9-b26c65a90696-kube-api-access-gtwzg\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:46:06.423561 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.423523 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf"] Oct 30 16:46:06.430564 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:06.430527 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-57bcc6d666-d8nkf"] Oct 30 16:46:07.329112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:07.329076 4861 generic.go:356] "Generic (PLEG): container finished" podID="7409c5f8-d599-4876-be1a-f5b0a25c0416" containerID="b18f190561fb3f4d2e9d115f847faa32d71712b3c3781dca6fa74f38b155542e" exitCode=0 Oct 30 16:46:07.329627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:07.329152 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" event={"ID":"7409c5f8-d599-4876-be1a-f5b0a25c0416","Type":"ContainerDied","Data":"b18f190561fb3f4d2e9d115f847faa32d71712b3c3781dca6fa74f38b155542e"} Oct 30 16:46:07.329742 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:07.329626 4861 scope.go:122] "RemoveContainer" containerID="b18f190561fb3f4d2e9d115f847faa32d71712b3c3781dca6fa74f38b155542e" Oct 30 16:46:07.331557 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:07.331521 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:46:07.331649 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:07.331611 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:46:07.331693 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:07.331659 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:46:07.808396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:07.808350 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74bf6688-428d-49ac-b1a9-b26c65a90696" path="/var/lib/kubelet/pods/74bf6688-428d-49ac-b1a9-b26c65a90696/volumes" Oct 30 16:46:07.808844 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:07.808824 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bdc5cff-5719-402a-97a2-c04cbf2d7c23" path="/var/lib/kubelet/pods/8bdc5cff-5719-402a-97a2-c04cbf2d7c23/volumes" Oct 30 16:46:07.821679 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:07.821652 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf"] Oct 30 16:46:07.855433 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:07.855406 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r"] Oct 30 16:46:07.888861 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:07.888834 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:08.038014 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:08.037980 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:08.172598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:08.172517 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:08.348027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:08.347992 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" event={"ID":"7409c5f8-d599-4876-be1a-f5b0a25c0416","Type":"ContainerStarted","Data":"93c17f851b02924708e3049a38af1af69a4ab802dfd9f3c2e202010926b16021"} Oct 30 16:46:08.987948 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:08.987917 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:46:09.057809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:09.057780 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:46:09.059605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:09.059578 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:46:09.059778 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:09.059640 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" podUID="7409c5f8-d599-4876-be1a-f5b0a25c0416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 16:46:09.356652 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:09.356619 4861 generic.go:356] "Generic (PLEG): container finished" podID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" containerID="1107a716eb0df91b1649fa75a12456e4914cba6a0e27b9c9fe729450f876b864" exitCode=0 Oct 30 16:46:09.357089 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:09.356696 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" event={"ID":"68fec9d9-15ed-45c2-95d3-fce8d047a6db","Type":"ContainerDied","Data":"1107a716eb0df91b1649fa75a12456e4914cba6a0e27b9c9fe729450f876b864"} Oct 30 16:46:09.357219 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:09.357202 4861 scope.go:122] "RemoveContainer" containerID="1107a716eb0df91b1649fa75a12456e4914cba6a0e27b9c9fe729450f876b864" Oct 30 16:46:09.359012 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:09.358972 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:46:09.359142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:09.359070 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:46:09.359142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:09.359122 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:46:10.057887 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:10.057852 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:46:10.058071 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:10.057919 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" podUID="7409c5f8-d599-4876-be1a-f5b0a25c0416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 16:46:10.239333 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:10.239301 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:10.490707 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:10.490660 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" event={"ID":"68fec9d9-15ed-45c2-95d3-fce8d047a6db","Type":"ContainerStarted","Data":"6847f4e934c31493a57f35bdb55ed2b2f3381427976215e2cf46cfbed9317c25"} Oct 30 16:46:10.895789 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:10.895678 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:46:10.897267 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:10.897236 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 16:46:10.897423 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:10.897292 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" podUID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 16:46:11.060747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:11.060669 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:46:11.060960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:11.060762 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" podUID="7409c5f8-d599-4876-be1a-f5b0a25c0416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 16:46:11.895947 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:11.895918 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 16:46:11.896322 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:11.895970 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" podUID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 16:46:12.076201 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:12.076167 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:46:12.076487 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:12.076461 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" podUID="7409c5f8-d599-4876-be1a-f5b0a25c0416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 16:46:12.823747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:12.823243 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" podUID="b8af0250-b0ec-4590-89c4-3a0ece7ac780" containerName="payload-processing" containerID="cri-o://1ba0aff15e97976aa0cc1480c028dec065ba846b712edb75f51d84caa73efe98" gracePeriod=25 Oct 30 16:46:12.871457 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:12.870355 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" podUID="97b5bbf9-35cf-48e8-9419-5a1eb03dacfe" containerName="payload-pre-processing" containerID="cri-o://e17ae1479a0c053430833412616d76f28dd95a29a6454238379ce1ceb9420002" gracePeriod=25 Oct 30 16:46:12.895922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:12.895357 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 16:46:12.895922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:12.895425 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" podUID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 16:46:13.059295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.059228 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:46:13.059583 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.059524 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" podUID="7409c5f8-d599-4876-be1a-f5b0a25c0416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 16:46:13.405347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.405262 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl"] Oct 30 16:46:13.415750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.406598 4861 cpu_manager.go:414] "containerMap: removing container" podUID="8bdc5cff-5719-402a-97a2-c04cbf2d7c23" containerName="payload-processing" Oct 30 16:46:13.415980 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.415957 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bdc5cff-5719-402a-97a2-c04cbf2d7c23" containerName="payload-processing" Oct 30 16:46:13.416152 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.416138 4861 cpu_manager.go:414] "containerMap: removing container" podUID="74bf6688-428d-49ac-b1a9-b26c65a90696" containerName="payload-pre-processing" Oct 30 16:46:13.416250 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.416238 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74bf6688-428d-49ac-b1a9-b26c65a90696" containerName="payload-pre-processing" Oct 30 16:46:13.416550 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.416529 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bdc5cff-5719-402a-97a2-c04cbf2d7c23" containerName="payload-processing" Oct 30 16:46:13.416685 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.416672 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="74bf6688-428d-49ac-b1a9-b26c65a90696" containerName="payload-pre-processing" Oct 30 16:46:13.852791 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.800597 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:13.852791 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.836545 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl"] Oct 30 16:46:13.858073 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.855749 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:46:13.885097 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.885068 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-06ac18-serving-cert\"" Oct 30 16:46:13.899254 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.897386 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 16:46:13.899254 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.897454 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" podUID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 16:46:13.983192 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.978847 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl\" (UID: \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:46:13.983192 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.978954 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl\" (UID: \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:46:13.983192 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.979238 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8qvd\" (UniqueName: \"kubernetes.io/projected/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-kube-api-access-n8qvd\") pod \"maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl\" (UID: \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:46:13.983192 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.979305 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-metrics-certs\") pod \"maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl\" (UID: \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:46:13.983192 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.981225 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:13.983192 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.982404 4861 generic.go:356] "Generic (PLEG): container finished" podID="97b5bbf9-35cf-48e8-9419-5a1eb03dacfe" containerID="e17ae1479a0c053430833412616d76f28dd95a29a6454238379ce1ceb9420002" exitCode=0 Oct 30 16:46:13.986486 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:13.986277 4861 generic.go:356] "Generic (PLEG): container finished" podID="b8af0250-b0ec-4590-89c4-3a0ece7ac780" containerID="1ba0aff15e97976aa0cc1480c028dec065ba846b712edb75f51d84caa73efe98" exitCode=0 Oct 30 16:46:14.060145 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.059145 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" event={"ID":"97b5bbf9-35cf-48e8-9419-5a1eb03dacfe","Type":"ContainerDied","Data":"e17ae1479a0c053430833412616d76f28dd95a29a6454238379ce1ceb9420002"} Oct 30 16:46:14.077419 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.077379 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" event={"ID":"b8af0250-b0ec-4590-89c4-3a0ece7ac780","Type":"ContainerDied","Data":"1ba0aff15e97976aa0cc1480c028dec065ba846b712edb75f51d84caa73efe98"} Oct 30 16:46:14.077877 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.075684 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 16:46:14.077966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.077915 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" podUID="7409c5f8-d599-4876-be1a-f5b0a25c0416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 16:46:14.084178 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.084150 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl\" (UID: \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:46:14.084549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.084530 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8qvd\" (UniqueName: \"kubernetes.io/projected/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-kube-api-access-n8qvd\") pod \"maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl\" (UID: \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:46:14.088599 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.087642 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-maas-api-tls\") pod \"maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl\" (UID: \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:46:14.088764 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.088747 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-metrics-certs\") pod \"maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl\" (UID: \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:46:14.090108 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.090055 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl\" (UID: \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:46:14.090214 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.090104 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl\" (UID: \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:46:14.093643 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.093617 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-metrics-certs\") pod \"maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl\" (UID: \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:46:14.174771 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.174715 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8qvd\" (UniqueName: \"kubernetes.io/projected/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-kube-api-access-n8qvd\") pod \"maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl\" (UID: \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:46:14.243995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.243961 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:46:14.290750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.289703 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" Oct 30 16:46:14.291974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.291952 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" Oct 30 16:46:14.404836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.395252 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8af0250-b0ec-4590-89c4-3a0ece7ac780-kube-api-access-xcsk8\" (UniqueName: \"kubernetes.io/projected/b8af0250-b0ec-4590-89c4-3a0ece7ac780-kube-api-access-xcsk8\") pod \"b8af0250-b0ec-4590-89c4-3a0ece7ac780\" (UID: \"b8af0250-b0ec-4590-89c4-3a0ece7ac780\") " Oct 30 16:46:14.404836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.395458 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b5bbf9-35cf-48e8-9419-5a1eb03dacfe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b5bbf9-35cf-48e8-9419-5a1eb03dacfe-plugins-config-volume\") pod \"97b5bbf9-35cf-48e8-9419-5a1eb03dacfe\" (UID: \"97b5bbf9-35cf-48e8-9419-5a1eb03dacfe\") " Oct 30 16:46:14.404836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.395492 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8af0250-b0ec-4590-89c4-3a0ece7ac780-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8af0250-b0ec-4590-89c4-3a0ece7ac780-plugins-config-volume\") pod \"b8af0250-b0ec-4590-89c4-3a0ece7ac780\" (UID: \"b8af0250-b0ec-4590-89c4-3a0ece7ac780\") " Oct 30 16:46:14.404836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.395526 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b5bbf9-35cf-48e8-9419-5a1eb03dacfe-kube-api-access-bglw7\" (UniqueName: \"kubernetes.io/projected/97b5bbf9-35cf-48e8-9419-5a1eb03dacfe-kube-api-access-bglw7\") pod \"97b5bbf9-35cf-48e8-9419-5a1eb03dacfe\" (UID: \"97b5bbf9-35cf-48e8-9419-5a1eb03dacfe\") " Oct 30 16:46:14.404836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.397345 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b5bbf9-35cf-48e8-9419-5a1eb03dacfe-plugins-config-volume" pod "97b5bbf9-35cf-48e8-9419-5a1eb03dacfe" (UID: "97b5bbf9-35cf-48e8-9419-5a1eb03dacfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:14.549749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.462967 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b5bbf9-35cf-48e8-9419-5a1eb03dacfe-kube-api-access-bglw7" pod "97b5bbf9-35cf-48e8-9419-5a1eb03dacfe" (UID: "97b5bbf9-35cf-48e8-9419-5a1eb03dacfe"). InnerVolumeSpecName "kube-api-access-bglw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:14.549749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.540796 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8af0250-b0ec-4590-89c4-3a0ece7ac780-kube-api-access-xcsk8" pod "b8af0250-b0ec-4590-89c4-3a0ece7ac780" (UID: "b8af0250-b0ec-4590-89c4-3a0ece7ac780"). InnerVolumeSpecName "kube-api-access-xcsk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:14.550007 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.549912 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bglw7\" (UniqueName: \"kubernetes.io/projected/97b5bbf9-35cf-48e8-9419-5a1eb03dacfe-kube-api-access-bglw7\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:46:14.550007 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.549942 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcsk8\" (UniqueName: \"kubernetes.io/projected/b8af0250-b0ec-4590-89c4-3a0ece7ac780-kube-api-access-xcsk8\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:46:14.550007 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.549959 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b5bbf9-35cf-48e8-9419-5a1eb03dacfe-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:46:14.569495 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.569000 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8af0250-b0ec-4590-89c4-3a0ece7ac780-plugins-config-volume" pod "b8af0250-b0ec-4590-89c4-3a0ece7ac780" (UID: "b8af0250-b0ec-4590-89c4-3a0ece7ac780"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:14.658799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.652918 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8af0250-b0ec-4590-89c4-3a0ece7ac780-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:46:14.966758 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.943128 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 16:46:14.966758 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.943188 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" podUID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 16:46:14.970908 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.969939 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl"] Oct 30 16:46:14.997355 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.997324 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" Oct 30 16:46:14.997355 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.997341 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r" event={"ID":"97b5bbf9-35cf-48e8-9419-5a1eb03dacfe","Type":"ContainerDied","Data":"88c9c81f739c46c03fcf2f8dad61aa46ab2bac118aa676db4564f9f0f443e141"} Oct 30 16:46:14.997532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:14.997404 4861 scope.go:122] "RemoveContainer" containerID="e17ae1479a0c053430833412616d76f28dd95a29a6454238379ce1ceb9420002" Oct 30 16:46:15.000254 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.000078 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" Oct 30 16:46:15.000254 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.000107 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf" event={"ID":"b8af0250-b0ec-4590-89c4-3a0ece7ac780","Type":"ContainerDied","Data":"511bea7941f246daab9c920968cbc9725f9754c01403d88ddd8ecf57f241696f"} Oct 30 16:46:15.058766 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.057590 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" event={"ID":"5fa04564-6eb3-4184-b9e2-3d64ac6752ad","Type":"ContainerStarted","Data":"3bb473b27309ac6a664541ccdebdbf9d98827a15314c98a0c9cb2b7f6f626437"} Oct 30 16:46:15.063182 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.063157 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:46:15.064527 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.064494 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:46:15.065868 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.065837 4861 scope.go:122] "RemoveContainer" containerID="1ba0aff15e97976aa0cc1480c028dec065ba846b712edb75f51d84caa73efe98" Oct 30 16:46:15.151491 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.150818 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r"] Oct 30 16:46:15.162013 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.161974 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:46:15.164143 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.164117 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-64c409-68849cb99f-l6n4r"] Oct 30 16:46:15.175807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.175779 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf"] Oct 30 16:46:15.188942 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.188918 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-64c409-59b8bf7c94-l6mhf"] Oct 30 16:46:15.194951 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.194893 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:15.859552 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.859475 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b5bbf9-35cf-48e8-9419-5a1eb03dacfe" path="/var/lib/kubelet/pods/97b5bbf9-35cf-48e8-9419-5a1eb03dacfe/volumes" Oct 30 16:46:15.859926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.859909 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8af0250-b0ec-4590-89c4-3a0ece7ac780" path="/var/lib/kubelet/pods/b8af0250-b0ec-4590-89c4-3a0ece7ac780/volumes" Oct 30 16:46:15.887676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.887649 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:15.897071 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.897049 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Oct 30 16:46:15.897247 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:15.897231 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" podUID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Oct 30 16:46:16.070055 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:16.070011 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" event={"ID":"5fa04564-6eb3-4184-b9e2-3d64ac6752ad","Type":"ContainerStarted","Data":"759c27d3db6949410211857f0c4ef5b3f1a7ee69b1fcb5f4a30f1b89d02f7958"} Oct 30 16:46:16.900403 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:16.900377 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:46:16.901628 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:16.901602 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:46:16.974173 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:16.974113 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" podStartSLOduration=3.463576677 podStartE2EDuration="3.974095696s" totalImagesPullingTime="510.519019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:16.127296414 +0000 UTC m=+1406.469970452" watchObservedRunningTime="2026-10-30 16:46:16.974095696 +0000 UTC m=+1407.316769735" Oct 30 16:46:17.077553 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:17.077526 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:46:18.135601 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:18.135384 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:18.223890 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:18.223851 4861 patch_prober.go:29] interesting pod/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:46:18.224056 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:18.223908 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" podUID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:46:20.103684 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:20.103612 4861 generic.go:356] "Generic (PLEG): container finished" podID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerID="1e530304e02b6027c5a5de5fdb235e248ddc6011b5b5418b29087724ac1ad666" exitCode=0 Oct 30 16:46:20.104089 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:20.103691 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" event={"ID":"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e","Type":"ContainerDied","Data":"1e530304e02b6027c5a5de5fdb235e248ddc6011b5b5418b29087724ac1ad666"} Oct 30 16:46:20.104137 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:20.104114 4861 scope.go:122] "RemoveContainer" containerID="1e530304e02b6027c5a5de5fdb235e248ddc6011b5b5418b29087724ac1ad666" Oct 30 16:46:20.105707 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:20.105680 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:46:20.105830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:20.105766 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:46:20.105830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:20.105805 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:46:20.299665 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:46:20.299613 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:46:20.503621 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:20.503585 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:21.125565 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:21.125529 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" event={"ID":"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e","Type":"ContainerStarted","Data":"7d8d7860238c2b67ddaa3cbdb398a6c903cfc749be5889c0b9b7e55caca6b26f"} Oct 30 16:46:21.128741 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:21.128696 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:46:21.129877 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:21.129853 4861 patch_prober.go:29] interesting pod/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 16:46:21.130086 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:21.130062 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" podUID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 16:46:22.129451 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:22.129417 4861 patch_prober.go:29] interesting pod/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 16:46:22.130111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:22.130080 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" podUID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 16:46:22.334262 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:22.334227 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:23.090598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:23.090568 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:46:23.127219 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:23.127181 4861 patch_prober.go:29] interesting pod/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 16:46:23.127397 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:23.127243 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" podUID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 16:46:24.128750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:24.127804 4861 patch_prober.go:29] interesting pod/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 16:46:24.128750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:24.127881 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" podUID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 16:46:24.232919 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:24.232881 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:25.127641 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:25.127604 4861 patch_prober.go:29] interesting pod/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 16:46:25.127869 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:25.127666 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" podUID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 16:46:26.063310 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:26.063276 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:26.144759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:26.131518 4861 patch_prober.go:29] interesting pod/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 16:46:26.144759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:26.131596 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" podUID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 16:46:27.131825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:27.131776 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:46:27.132333 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:27.132209 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:46:27.132810 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:27.132789 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:46:27.356586 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:27.356555 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:27.497372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:27.497316 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:28.744287 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:28.744221 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:29.981551 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:29.981519 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:30.163151 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.162907 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:30.453917 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.453829 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc"] Oct 30 16:46:30.455167 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.455124 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b8af0250-b0ec-4590-89c4-3a0ece7ac780" containerName="payload-processing" Oct 30 16:46:30.455355 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.455339 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8af0250-b0ec-4590-89c4-3a0ece7ac780" containerName="payload-processing" Oct 30 16:46:30.458976 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.458952 4861 cpu_manager.go:414] "containerMap: removing container" podUID="97b5bbf9-35cf-48e8-9419-5a1eb03dacfe" containerName="payload-pre-processing" Oct 30 16:46:30.459154 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.459141 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b5bbf9-35cf-48e8-9419-5a1eb03dacfe" containerName="payload-pre-processing" Oct 30 16:46:30.459457 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.459442 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b5bbf9-35cf-48e8-9419-5a1eb03dacfe" containerName="payload-pre-processing" Oct 30 16:46:30.462795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.462777 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8af0250-b0ec-4590-89c4-3a0ece7ac780" containerName="payload-processing" Oct 30 16:46:30.525831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.525798 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" Oct 30 16:46:30.532604 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.532575 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc"] Oct 30 16:46:30.541868 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.541835 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp"] Oct 30 16:46:30.547682 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.547658 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" Oct 30 16:46:30.556913 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.556889 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-dc1ea0-dockercfg-4xnhr\"" Oct 30 16:46:30.557894 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.557871 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-dc1ea0\"" Oct 30 16:46:30.563573 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.563553 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp"] Oct 30 16:46:30.622423 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.622388 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e0283d9-a9fb-40a1-873a-d66b923ee500-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp\" (UID: \"7e0283d9-a9fb-40a1-873a-d66b923ee500\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" Oct 30 16:46:30.622619 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.622552 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5159a2ef-ff89-4345-bcd5-b76bb3b6aec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc\" (UID: \"5159a2ef-ff89-4345-bcd5-b76bb3b6aec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" Oct 30 16:46:30.622764 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.622711 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mkjl\" (UniqueName: \"kubernetes.io/projected/5159a2ef-ff89-4345-bcd5-b76bb3b6aec9-kube-api-access-5mkjl\") pod \"payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc\" (UID: \"5159a2ef-ff89-4345-bcd5-b76bb3b6aec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" Oct 30 16:46:30.622863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.622779 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5xlh\" (UniqueName: \"kubernetes.io/projected/7e0283d9-a9fb-40a1-873a-d66b923ee500-kube-api-access-t5xlh\") pod \"payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp\" (UID: \"7e0283d9-a9fb-40a1-873a-d66b923ee500\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" Oct 30 16:46:30.723769 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.723699 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mkjl\" (UniqueName: \"kubernetes.io/projected/5159a2ef-ff89-4345-bcd5-b76bb3b6aec9-kube-api-access-5mkjl\") pod \"payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc\" (UID: \"5159a2ef-ff89-4345-bcd5-b76bb3b6aec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" Oct 30 16:46:30.723966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.723787 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5xlh\" (UniqueName: \"kubernetes.io/projected/7e0283d9-a9fb-40a1-873a-d66b923ee500-kube-api-access-t5xlh\") pod \"payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp\" (UID: \"7e0283d9-a9fb-40a1-873a-d66b923ee500\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" Oct 30 16:46:30.723966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.723898 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e0283d9-a9fb-40a1-873a-d66b923ee500-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp\" (UID: \"7e0283d9-a9fb-40a1-873a-d66b923ee500\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" Oct 30 16:46:30.724121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.724040 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5159a2ef-ff89-4345-bcd5-b76bb3b6aec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc\" (UID: \"5159a2ef-ff89-4345-bcd5-b76bb3b6aec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" Oct 30 16:46:30.724833 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.724815 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e0283d9-a9fb-40a1-873a-d66b923ee500-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp\" (UID: \"7e0283d9-a9fb-40a1-873a-d66b923ee500\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" Oct 30 16:46:30.724927 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.724833 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5159a2ef-ff89-4345-bcd5-b76bb3b6aec9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc\" (UID: \"5159a2ef-ff89-4345-bcd5-b76bb3b6aec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" Oct 30 16:46:30.748254 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.748228 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mkjl\" (UniqueName: \"kubernetes.io/projected/5159a2ef-ff89-4345-bcd5-b76bb3b6aec9-kube-api-access-5mkjl\") pod \"payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc\" (UID: \"5159a2ef-ff89-4345-bcd5-b76bb3b6aec9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" Oct 30 16:46:30.748440 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.748414 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5xlh\" (UniqueName: \"kubernetes.io/projected/7e0283d9-a9fb-40a1-873a-d66b923ee500-kube-api-access-t5xlh\") pod \"payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp\" (UID: \"7e0283d9-a9fb-40a1-873a-d66b923ee500\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" Oct 30 16:46:30.855130 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.855104 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" Oct 30 16:46:30.875766 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.874835 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" Oct 30 16:46:30.979686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:30.979449 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:31.051022 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.050972 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7"] Oct 30 16:46:31.054417 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.054381 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" podUID="2e07cfe3-baff-49e7-8bf7-73715fa20049" containerName="payload-pre-processing" containerID="cri-o://613ce6efe31f66a6cf3fa0e1445a1bae6880deb4441dca7565074986eb238d23" gracePeriod=30 Oct 30 16:46:31.141279 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.141240 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:31.207762 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.199052 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc"] Oct 30 16:46:31.214550 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.214518 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp"] Oct 30 16:46:31.223823 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:46:31.223702 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0283d9_a9fb_40a1_873a_d66b923ee500.slice/crio-ec5ea82083ef3ea1a1de6094cac1647c06f0b444f0c0e20e6280c90cd92cee45 WatchSource:0}: Error finding container ec5ea82083ef3ea1a1de6094cac1647c06f0b444f0c0e20e6280c90cd92cee45: Status 404 returned error can't find the container with id ec5ea82083ef3ea1a1de6094cac1647c06f0b444f0c0e20e6280c90cd92cee45 Oct 30 16:46:31.238228 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.238203 4861 generic.go:356] "Generic (PLEG): container finished" podID="2e07cfe3-baff-49e7-8bf7-73715fa20049" containerID="613ce6efe31f66a6cf3fa0e1445a1bae6880deb4441dca7565074986eb238d23" exitCode=0 Oct 30 16:46:31.238359 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.238307 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" event={"ID":"2e07cfe3-baff-49e7-8bf7-73715fa20049","Type":"ContainerDied","Data":"613ce6efe31f66a6cf3fa0e1445a1bae6880deb4441dca7565074986eb238d23"} Oct 30 16:46:31.241289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.241257 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" event={"ID":"7e0283d9-a9fb-40a1-873a-d66b923ee500","Type":"ContainerStarted","Data":"ec5ea82083ef3ea1a1de6094cac1647c06f0b444f0c0e20e6280c90cd92cee45"} Oct 30 16:46:31.243319 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.243293 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" event={"ID":"5159a2ef-ff89-4345-bcd5-b76bb3b6aec9","Type":"ContainerStarted","Data":"9e5f922a3df4c5c7a06ad85936f16aa7de38c3282c89765bbeaf8fade4ffd4ef"} Oct 30 16:46:31.292052 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.292011 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:31.346205 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.346184 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" Oct 30 16:46:31.435752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.434495 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e07cfe3-baff-49e7-8bf7-73715fa20049-kube-api-access-zbttf\" (UniqueName: \"kubernetes.io/projected/2e07cfe3-baff-49e7-8bf7-73715fa20049-kube-api-access-zbttf\") pod \"2e07cfe3-baff-49e7-8bf7-73715fa20049\" (UID: \"2e07cfe3-baff-49e7-8bf7-73715fa20049\") " Oct 30 16:46:31.435752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.434576 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e07cfe3-baff-49e7-8bf7-73715fa20049-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e07cfe3-baff-49e7-8bf7-73715fa20049-plugins-config-volume\") pod \"2e07cfe3-baff-49e7-8bf7-73715fa20049\" (UID: \"2e07cfe3-baff-49e7-8bf7-73715fa20049\") " Oct 30 16:46:31.442671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.436040 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e07cfe3-baff-49e7-8bf7-73715fa20049-plugins-config-volume" pod "2e07cfe3-baff-49e7-8bf7-73715fa20049" (UID: "2e07cfe3-baff-49e7-8bf7-73715fa20049"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:31.443556 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.443523 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e07cfe3-baff-49e7-8bf7-73715fa20049-kube-api-access-zbttf" pod "2e07cfe3-baff-49e7-8bf7-73715fa20049" (UID: "2e07cfe3-baff-49e7-8bf7-73715fa20049"). InnerVolumeSpecName "kube-api-access-zbttf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:31.546833 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.544940 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbttf\" (UniqueName: \"kubernetes.io/projected/2e07cfe3-baff-49e7-8bf7-73715fa20049-kube-api-access-zbttf\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:46:31.546833 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.544980 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e07cfe3-baff-49e7-8bf7-73715fa20049-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:46:31.635785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.595348 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:31.847331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.847301 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:31.994515 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:31.994482 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:32.255550 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:32.255508 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" event={"ID":"5159a2ef-ff89-4345-bcd5-b76bb3b6aec9","Type":"ContainerStarted","Data":"7f279cebe680bf812d515a0214681381604ed1748a345d47d10dbdb1c6329e25"} Oct 30 16:46:32.256046 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:32.255857 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" Oct 30 16:46:32.257166 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:32.257138 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" Oct 30 16:46:32.257325 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:32.257180 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7" event={"ID":"2e07cfe3-baff-49e7-8bf7-73715fa20049","Type":"ContainerDied","Data":"89ae89003adcd2c9941c858c4b3f9f7307301cac2c7b48bafcd8ea6dd21d61fd"} Oct 30 16:46:32.257325 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:32.257233 4861 scope.go:122] "RemoveContainer" containerID="613ce6efe31f66a6cf3fa0e1445a1bae6880deb4441dca7565074986eb238d23" Oct 30 16:46:32.259517 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:32.259484 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" event={"ID":"7e0283d9-a9fb-40a1-873a-d66b923ee500","Type":"ContainerStarted","Data":"4de093a12900c35d31282f964a2d86decb9ff34c43f3310edb8461cb9eba722b"} Oct 30 16:46:32.259652 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:32.259569 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" Oct 30 16:46:32.300466 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:32.300424 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" podStartSLOduration=2.300411554 podStartE2EDuration="2.300411554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:32.296716771 +0000 UTC m=+1422.639390809" watchObservedRunningTime="2026-10-30 16:46:32.300411554 +0000 UTC m=+1422.643085592" Oct 30 16:46:32.342895 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:32.342872 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7"] Oct 30 16:46:32.346273 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:32.346242 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:32.348193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:32.348170 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0a98d1-6d866557d7-l5vp7"] Oct 30 16:46:32.389415 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:32.389363 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" podStartSLOduration=2.389347884 podStartE2EDuration="2.389347884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:32.385935253 +0000 UTC m=+1422.728609289" watchObservedRunningTime="2026-10-30 16:46:32.389347884 +0000 UTC m=+1422.732021924" Oct 30 16:46:33.808484 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:33.808451 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e07cfe3-baff-49e7-8bf7-73715fa20049" path="/var/lib/kubelet/pods/2e07cfe3-baff-49e7-8bf7-73715fa20049/volumes" Oct 30 16:46:34.335855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:34.335823 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:36.373780 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:36.368831 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:37.629256 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:37.629228 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:37.788441 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:37.788372 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:38.819748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:38.816645 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:39.892536 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:39.892499 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:40.042089 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:40.042052 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:40.226211 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:40.226176 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:40.801001 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:40.800969 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:42.832060 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:42.832030 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:43.270397 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:43.270364 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" Oct 30 16:46:43.270583 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:43.270522 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" Oct 30 16:46:43.524150 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:43.524065 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:45.291937 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:45.291905 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:47.227053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:47.227018 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:49.115372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:49.115339 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:50.291343 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:46:50.291305 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:46:50.450102 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:50.450077 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22"] Oct 30 16:46:50.629831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:50.629706 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:50.785188 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:50.785152 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:50.950800 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:50.949040 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:51.726060 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:51.726024 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:52.215238 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:52.215164 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:52.353168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:52.353138 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:53.595561 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:53.595527 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:55.450750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:55.450685 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" podUID="6ecf9d2f-4389-46e8-801d-970f54fe8a14" containerName="payload-processing" containerID="cri-o://2669575e75923b99d4d2a4012a18a9298955b00e0dcbdd316527f7e2a3507766" gracePeriod=25 Oct 30 16:46:55.499146 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:55.499120 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:55.762418 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:55.762390 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" Oct 30 16:46:55.920031 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:55.920003 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ecf9d2f-4389-46e8-801d-970f54fe8a14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecf9d2f-4389-46e8-801d-970f54fe8a14-plugins-config-volume\") pod \"6ecf9d2f-4389-46e8-801d-970f54fe8a14\" (UID: \"6ecf9d2f-4389-46e8-801d-970f54fe8a14\") " Oct 30 16:46:55.920208 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:55.920120 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ecf9d2f-4389-46e8-801d-970f54fe8a14-kube-api-access-795wt\" (UniqueName: \"kubernetes.io/projected/6ecf9d2f-4389-46e8-801d-970f54fe8a14-kube-api-access-795wt\") pod \"6ecf9d2f-4389-46e8-801d-970f54fe8a14\" (UID: \"6ecf9d2f-4389-46e8-801d-970f54fe8a14\") " Oct 30 16:46:55.920353 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:55.920331 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ecf9d2f-4389-46e8-801d-970f54fe8a14-plugins-config-volume" pod "6ecf9d2f-4389-46e8-801d-970f54fe8a14" (UID: "6ecf9d2f-4389-46e8-801d-970f54fe8a14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:46:55.920650 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:55.920636 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecf9d2f-4389-46e8-801d-970f54fe8a14-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:46:55.922352 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:55.922332 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecf9d2f-4389-46e8-801d-970f54fe8a14-kube-api-access-795wt" pod "6ecf9d2f-4389-46e8-801d-970f54fe8a14" (UID: "6ecf9d2f-4389-46e8-801d-970f54fe8a14"). InnerVolumeSpecName "kube-api-access-795wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:46:56.021974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:56.021894 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-795wt\" (UniqueName: \"kubernetes.io/projected/6ecf9d2f-4389-46e8-801d-970f54fe8a14-kube-api-access-795wt\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:46:56.474824 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:56.474785 4861 generic.go:356] "Generic (PLEG): container finished" podID="6ecf9d2f-4389-46e8-801d-970f54fe8a14" containerID="2669575e75923b99d4d2a4012a18a9298955b00e0dcbdd316527f7e2a3507766" exitCode=0 Oct 30 16:46:56.475313 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:56.474829 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" Oct 30 16:46:56.475313 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:56.474863 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" event={"ID":"6ecf9d2f-4389-46e8-801d-970f54fe8a14","Type":"ContainerDied","Data":"2669575e75923b99d4d2a4012a18a9298955b00e0dcbdd316527f7e2a3507766"} Oct 30 16:46:56.475313 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:56.474930 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22" event={"ID":"6ecf9d2f-4389-46e8-801d-970f54fe8a14","Type":"ContainerDied","Data":"cf80523a03d91abda60352c688160c2cdd35182d9451df54e0befa94fff2847c"} Oct 30 16:46:56.475313 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:56.474962 4861 scope.go:122] "RemoveContainer" containerID="2669575e75923b99d4d2a4012a18a9298955b00e0dcbdd316527f7e2a3507766" Oct 30 16:46:56.486475 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:56.486454 4861 scope.go:122] "RemoveContainer" containerID="2669575e75923b99d4d2a4012a18a9298955b00e0dcbdd316527f7e2a3507766" Oct 30 16:46:56.486760 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:46:56.486707 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2669575e75923b99d4d2a4012a18a9298955b00e0dcbdd316527f7e2a3507766\": container with ID starting with 2669575e75923b99d4d2a4012a18a9298955b00e0dcbdd316527f7e2a3507766 not found: ID does not exist" containerID="2669575e75923b99d4d2a4012a18a9298955b00e0dcbdd316527f7e2a3507766" Oct 30 16:46:56.486841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:56.486752 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2669575e75923b99d4d2a4012a18a9298955b00e0dcbdd316527f7e2a3507766"} err="failed to get container status \"2669575e75923b99d4d2a4012a18a9298955b00e0dcbdd316527f7e2a3507766\": rpc error: code = NotFound desc = could not find container \"2669575e75923b99d4d2a4012a18a9298955b00e0dcbdd316527f7e2a3507766\": container with ID starting with 2669575e75923b99d4d2a4012a18a9298955b00e0dcbdd316527f7e2a3507766 not found: ID does not exist" Oct 30 16:46:56.534954 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:56.534932 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22"] Oct 30 16:46:56.547175 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:56.547152 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-06ac18-74c9588996-hlp22"] Oct 30 16:46:56.861108 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:56.861021 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:57.003583 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:57.003530 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:57.389136 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:57.389107 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:46:57.807981 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:57.807948 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ecf9d2f-4389-46e8-801d-970f54fe8a14" path="/var/lib/kubelet/pods/6ecf9d2f-4389-46e8-801d-970f54fe8a14/volumes" Oct 30 16:46:59.560485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:59.560451 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn"] Oct 30 16:46:59.560987 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:59.560768 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" podUID="45769419-34ec-42d9-b5de-035877d01d2b" containerName="maas-api" containerID="cri-o://139d755e9fa9fd8f15e5ca41f5cc1a6c6a383bb88d2dbd5d3b41dfcfbd16731e" gracePeriod=30 Oct 30 16:46:59.657143 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:46:59.657109 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:00.013779 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.011993 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:47:00.146773 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.146657 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45769419-34ec-42d9-b5de-035877d01d2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/45769419-34ec-42d9-b5de-035877d01d2b-metrics-certs\") pod \"45769419-34ec-42d9-b5de-035877d01d2b\" (UID: \"45769419-34ec-42d9-b5de-035877d01d2b\") " Oct 30 16:47:00.146773 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.146750 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45769419-34ec-42d9-b5de-035877d01d2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45769419-34ec-42d9-b5de-035877d01d2b-maas-api-tls\") pod \"45769419-34ec-42d9-b5de-035877d01d2b\" (UID: \"45769419-34ec-42d9-b5de-035877d01d2b\") " Oct 30 16:47:00.147053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.146952 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45769419-34ec-42d9-b5de-035877d01d2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45769419-34ec-42d9-b5de-035877d01d2b-openshift-service-ca\") pod \"45769419-34ec-42d9-b5de-035877d01d2b\" (UID: \"45769419-34ec-42d9-b5de-035877d01d2b\") " Oct 30 16:47:00.147053 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.146983 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45769419-34ec-42d9-b5de-035877d01d2b-kube-api-access-bs5cv\" (UniqueName: \"kubernetes.io/projected/45769419-34ec-42d9-b5de-035877d01d2b-kube-api-access-bs5cv\") pod \"45769419-34ec-42d9-b5de-035877d01d2b\" (UID: \"45769419-34ec-42d9-b5de-035877d01d2b\") " Oct 30 16:47:00.175660 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.169357 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45769419-34ec-42d9-b5de-035877d01d2b-openshift-service-ca" pod "45769419-34ec-42d9-b5de-035877d01d2b" (UID: "45769419-34ec-42d9-b5de-035877d01d2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:00.225561 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.216329 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45769419-34ec-42d9-b5de-035877d01d2b-maas-api-tls" pod "45769419-34ec-42d9-b5de-035877d01d2b" (UID: "45769419-34ec-42d9-b5de-035877d01d2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:00.225561 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.216465 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45769419-34ec-42d9-b5de-035877d01d2b-kube-api-access-bs5cv" pod "45769419-34ec-42d9-b5de-035877d01d2b" (UID: "45769419-34ec-42d9-b5de-035877d01d2b"). InnerVolumeSpecName "kube-api-access-bs5cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:00.225561 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.222249 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45769419-34ec-42d9-b5de-035877d01d2b-metrics-certs" pod "45769419-34ec-42d9-b5de-035877d01d2b" (UID: "45769419-34ec-42d9-b5de-035877d01d2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:00.249324 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.249288 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45769419-34ec-42d9-b5de-035877d01d2b-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:00.249324 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.249317 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45769419-34ec-42d9-b5de-035877d01d2b-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:00.249516 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.249330 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45769419-34ec-42d9-b5de-035877d01d2b-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:00.249516 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.249344 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs5cv\" (UniqueName: \"kubernetes.io/projected/45769419-34ec-42d9-b5de-035877d01d2b-kube-api-access-bs5cv\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:00.310189 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.310158 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:00.506745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.506683 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:00.506939 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.506907 4861 generic.go:356] "Generic (PLEG): container finished" podID="45769419-34ec-42d9-b5de-035877d01d2b" containerID="139d755e9fa9fd8f15e5ca41f5cc1a6c6a383bb88d2dbd5d3b41dfcfbd16731e" exitCode=0 Oct 30 16:47:00.507106 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.506961 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" Oct 30 16:47:00.507198 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.507176 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" event={"ID":"45769419-34ec-42d9-b5de-035877d01d2b","Type":"ContainerDied","Data":"139d755e9fa9fd8f15e5ca41f5cc1a6c6a383bb88d2dbd5d3b41dfcfbd16731e"} Oct 30 16:47:00.507272 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.507227 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn" event={"ID":"45769419-34ec-42d9-b5de-035877d01d2b","Type":"ContainerDied","Data":"b4c9b7f331f23149835f331bc62647b855ab264cea18df6d4664b981434125f5"} Oct 30 16:47:00.507272 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.507260 4861 scope.go:122] "RemoveContainer" containerID="139d755e9fa9fd8f15e5ca41f5cc1a6c6a383bb88d2dbd5d3b41dfcfbd16731e" Oct 30 16:47:00.552094 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.552066 4861 scope.go:122] "RemoveContainer" containerID="139d755e9fa9fd8f15e5ca41f5cc1a6c6a383bb88d2dbd5d3b41dfcfbd16731e" Oct 30 16:47:00.552492 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:00.552465 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139d755e9fa9fd8f15e5ca41f5cc1a6c6a383bb88d2dbd5d3b41dfcfbd16731e\": container with ID starting with 139d755e9fa9fd8f15e5ca41f5cc1a6c6a383bb88d2dbd5d3b41dfcfbd16731e not found: ID does not exist" containerID="139d755e9fa9fd8f15e5ca41f5cc1a6c6a383bb88d2dbd5d3b41dfcfbd16731e" Oct 30 16:47:00.552778 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.552506 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139d755e9fa9fd8f15e5ca41f5cc1a6c6a383bb88d2dbd5d3b41dfcfbd16731e"} err="failed to get container status \"139d755e9fa9fd8f15e5ca41f5cc1a6c6a383bb88d2dbd5d3b41dfcfbd16731e\": rpc error: code = NotFound desc = could not find container \"139d755e9fa9fd8f15e5ca41f5cc1a6c6a383bb88d2dbd5d3b41dfcfbd16731e\": container with ID starting with 139d755e9fa9fd8f15e5ca41f5cc1a6c6a383bb88d2dbd5d3b41dfcfbd16731e not found: ID does not exist" Oct 30 16:47:00.604104 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.604075 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn"] Oct 30 16:47:00.609553 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.609515 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-08fcc8-65d548d46c-2gwkn"] Oct 30 16:47:00.685217 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:00.685184 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:01.055013 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:01.054972 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:01.808239 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:01.808210 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45769419-34ec-42d9-b5de-035877d01d2b" path="/var/lib/kubelet/pods/45769419-34ec-42d9-b5de-035877d01d2b/volumes" Oct 30 16:47:02.701219 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:02.701181 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w"] Oct 30 16:47:02.701413 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:02.701387 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" podUID="e88f463a-7f54-49ba-94da-f6522fe52aa2" containerName="maas-api" containerID="cri-o://b3be194a85186ff822573c74c43c39399c1d8906f29dc7b5bd42525a084548b6" gracePeriod=30 Oct 30 16:47:02.796213 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:02.796185 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:02.933113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:02.933076 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns"] Oct 30 16:47:02.933826 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:02.933337 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" podUID="5e75dc21-662f-4048-8455-df78d3db3b9a" containerName="payload-pre-processing" containerID="cri-o://04f3253cdeac22fda884e7560ce7f633be14abe64029d98076ccd688bd3a1509" gracePeriod=30 Oct 30 16:47:02.985960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:02.985915 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:03.075048 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.075014 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:47:03.132314 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.132284 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:03.197683 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.197650 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e88f463a-7f54-49ba-94da-f6522fe52aa2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e88f463a-7f54-49ba-94da-f6522fe52aa2-maas-api-tls\") pod \"e88f463a-7f54-49ba-94da-f6522fe52aa2\" (UID: \"e88f463a-7f54-49ba-94da-f6522fe52aa2\") " Oct 30 16:47:03.197865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.197765 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e88f463a-7f54-49ba-94da-f6522fe52aa2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e88f463a-7f54-49ba-94da-f6522fe52aa2-metrics-certs\") pod \"e88f463a-7f54-49ba-94da-f6522fe52aa2\" (UID: \"e88f463a-7f54-49ba-94da-f6522fe52aa2\") " Oct 30 16:47:03.197865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.197854 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88f463a-7f54-49ba-94da-f6522fe52aa2-kube-api-access-5sq5l\" (UniqueName: \"kubernetes.io/projected/e88f463a-7f54-49ba-94da-f6522fe52aa2-kube-api-access-5sq5l\") pod \"e88f463a-7f54-49ba-94da-f6522fe52aa2\" (UID: \"e88f463a-7f54-49ba-94da-f6522fe52aa2\") " Oct 30 16:47:03.198002 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.197887 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e88f463a-7f54-49ba-94da-f6522fe52aa2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e88f463a-7f54-49ba-94da-f6522fe52aa2-openshift-service-ca\") pod \"e88f463a-7f54-49ba-94da-f6522fe52aa2\" (UID: \"e88f463a-7f54-49ba-94da-f6522fe52aa2\") " Oct 30 16:47:03.198438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.198379 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e88f463a-7f54-49ba-94da-f6522fe52aa2-openshift-service-ca" pod "e88f463a-7f54-49ba-94da-f6522fe52aa2" (UID: "e88f463a-7f54-49ba-94da-f6522fe52aa2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:03.198546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.198520 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" Oct 30 16:47:03.198715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.198691 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e88f463a-7f54-49ba-94da-f6522fe52aa2-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:03.200430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.200402 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e88f463a-7f54-49ba-94da-f6522fe52aa2-metrics-certs" pod "e88f463a-7f54-49ba-94da-f6522fe52aa2" (UID: "e88f463a-7f54-49ba-94da-f6522fe52aa2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:03.200787 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.200716 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88f463a-7f54-49ba-94da-f6522fe52aa2-kube-api-access-5sq5l" pod "e88f463a-7f54-49ba-94da-f6522fe52aa2" (UID: "e88f463a-7f54-49ba-94da-f6522fe52aa2"). InnerVolumeSpecName "kube-api-access-5sq5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:03.201291 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.201276 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e88f463a-7f54-49ba-94da-f6522fe52aa2-maas-api-tls" pod "e88f463a-7f54-49ba-94da-f6522fe52aa2" (UID: "e88f463a-7f54-49ba-94da-f6522fe52aa2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:03.301891 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.301260 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e75dc21-662f-4048-8455-df78d3db3b9a-kube-api-access-d5q8j\" (UniqueName: \"kubernetes.io/projected/5e75dc21-662f-4048-8455-df78d3db3b9a-kube-api-access-d5q8j\") pod \"5e75dc21-662f-4048-8455-df78d3db3b9a\" (UID: \"5e75dc21-662f-4048-8455-df78d3db3b9a\") " Oct 30 16:47:03.301891 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.301507 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e75dc21-662f-4048-8455-df78d3db3b9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e75dc21-662f-4048-8455-df78d3db3b9a-plugins-config-volume\") pod \"5e75dc21-662f-4048-8455-df78d3db3b9a\" (UID: \"5e75dc21-662f-4048-8455-df78d3db3b9a\") " Oct 30 16:47:03.324496 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.316361 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e88f463a-7f54-49ba-94da-f6522fe52aa2-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:03.324496 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.316397 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e88f463a-7f54-49ba-94da-f6522fe52aa2-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:03.324496 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.316411 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sq5l\" (UniqueName: \"kubernetes.io/projected/e88f463a-7f54-49ba-94da-f6522fe52aa2-kube-api-access-5sq5l\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:03.341035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.340988 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e75dc21-662f-4048-8455-df78d3db3b9a-plugins-config-volume" pod "5e75dc21-662f-4048-8455-df78d3db3b9a" (UID: "5e75dc21-662f-4048-8455-df78d3db3b9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:03.422403 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.421079 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e75dc21-662f-4048-8455-df78d3db3b9a-kube-api-access-d5q8j" pod "5e75dc21-662f-4048-8455-df78d3db3b9a" (UID: "5e75dc21-662f-4048-8455-df78d3db3b9a"). InnerVolumeSpecName "kube-api-access-d5q8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:03.422403 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.421137 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:03.422403 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.422336 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e75dc21-662f-4048-8455-df78d3db3b9a-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:03.422403 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.422363 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5q8j\" (UniqueName: \"kubernetes.io/projected/5e75dc21-662f-4048-8455-df78d3db3b9a-kube-api-access-d5q8j\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:03.529305 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.529274 4861 generic.go:356] "Generic (PLEG): container finished" podID="e88f463a-7f54-49ba-94da-f6522fe52aa2" containerID="b3be194a85186ff822573c74c43c39399c1d8906f29dc7b5bd42525a084548b6" exitCode=0 Oct 30 16:47:03.529482 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.529340 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" Oct 30 16:47:03.529482 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.529362 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" event={"ID":"e88f463a-7f54-49ba-94da-f6522fe52aa2","Type":"ContainerDied","Data":"b3be194a85186ff822573c74c43c39399c1d8906f29dc7b5bd42525a084548b6"} Oct 30 16:47:03.529482 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.529436 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w" event={"ID":"e88f463a-7f54-49ba-94da-f6522fe52aa2","Type":"ContainerDied","Data":"5036c7dffd9b5b549b575be31cb621aa01897feb66fae604911045bc292e417a"} Oct 30 16:47:03.529482 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.529467 4861 scope.go:122] "RemoveContainer" containerID="b3be194a85186ff822573c74c43c39399c1d8906f29dc7b5bd42525a084548b6" Oct 30 16:47:03.530917 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.530885 4861 generic.go:356] "Generic (PLEG): container finished" podID="5e75dc21-662f-4048-8455-df78d3db3b9a" containerID="04f3253cdeac22fda884e7560ce7f633be14abe64029d98076ccd688bd3a1509" exitCode=0 Oct 30 16:47:03.531040 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.530919 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" event={"ID":"5e75dc21-662f-4048-8455-df78d3db3b9a","Type":"ContainerDied","Data":"04f3253cdeac22fda884e7560ce7f633be14abe64029d98076ccd688bd3a1509"} Oct 30 16:47:03.531040 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.530943 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" Oct 30 16:47:03.531040 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.530962 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns" event={"ID":"5e75dc21-662f-4048-8455-df78d3db3b9a","Type":"ContainerDied","Data":"d294bc08ae75f5645d88d9f5f63c8209987ee5b5bfb4f6f4177b7e2bec3417d3"} Oct 30 16:47:03.541337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.541315 4861 scope.go:122] "RemoveContainer" containerID="b3be194a85186ff822573c74c43c39399c1d8906f29dc7b5bd42525a084548b6" Oct 30 16:47:03.541667 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:03.541649 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3be194a85186ff822573c74c43c39399c1d8906f29dc7b5bd42525a084548b6\": container with ID starting with b3be194a85186ff822573c74c43c39399c1d8906f29dc7b5bd42525a084548b6 not found: ID does not exist" containerID="b3be194a85186ff822573c74c43c39399c1d8906f29dc7b5bd42525a084548b6" Oct 30 16:47:03.541715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.541675 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3be194a85186ff822573c74c43c39399c1d8906f29dc7b5bd42525a084548b6"} err="failed to get container status \"b3be194a85186ff822573c74c43c39399c1d8906f29dc7b5bd42525a084548b6\": rpc error: code = NotFound desc = could not find container \"b3be194a85186ff822573c74c43c39399c1d8906f29dc7b5bd42525a084548b6\": container with ID starting with b3be194a85186ff822573c74c43c39399c1d8906f29dc7b5bd42525a084548b6 not found: ID does not exist" Oct 30 16:47:03.541715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.541693 4861 scope.go:122] "RemoveContainer" containerID="04f3253cdeac22fda884e7560ce7f633be14abe64029d98076ccd688bd3a1509" Oct 30 16:47:03.554847 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.554822 4861 scope.go:122] "RemoveContainer" containerID="04f3253cdeac22fda884e7560ce7f633be14abe64029d98076ccd688bd3a1509" Oct 30 16:47:03.555149 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:03.555125 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04f3253cdeac22fda884e7560ce7f633be14abe64029d98076ccd688bd3a1509\": container with ID starting with 04f3253cdeac22fda884e7560ce7f633be14abe64029d98076ccd688bd3a1509 not found: ID does not exist" containerID="04f3253cdeac22fda884e7560ce7f633be14abe64029d98076ccd688bd3a1509" Oct 30 16:47:03.555209 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.555155 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04f3253cdeac22fda884e7560ce7f633be14abe64029d98076ccd688bd3a1509"} err="failed to get container status \"04f3253cdeac22fda884e7560ce7f633be14abe64029d98076ccd688bd3a1509\": rpc error: code = NotFound desc = could not find container \"04f3253cdeac22fda884e7560ce7f633be14abe64029d98076ccd688bd3a1509\": container with ID starting with 04f3253cdeac22fda884e7560ce7f633be14abe64029d98076ccd688bd3a1509 not found: ID does not exist" Oct 30 16:47:03.578399 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.578373 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:03.582620 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.582601 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns"] Oct 30 16:47:03.587197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.587177 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-843ab3-86f9674d7-hxnns"] Oct 30 16:47:03.684751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.679353 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w"] Oct 30 16:47:03.691653 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.691615 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-843ab3-768d6cd69c-8sq5w"] Oct 30 16:47:03.716753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.716709 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:03.808142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.808081 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e75dc21-662f-4048-8455-df78d3db3b9a" path="/var/lib/kubelet/pods/5e75dc21-662f-4048-8455-df78d3db3b9a/volumes" Oct 30 16:47:03.808548 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:03.808534 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e88f463a-7f54-49ba-94da-f6522fe52aa2" path="/var/lib/kubelet/pods/e88f463a-7f54-49ba-94da-f6522fe52aa2/volumes" Oct 30 16:47:04.076626 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:04.076541 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:05.656040 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:05.656005 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:07.691142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:07.691112 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:07.824671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:07.824637 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:08.050762 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:08.050694 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:08.200169 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:08.200130 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:10.146765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:10.143274 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:10.787334 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:10.787300 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:10.939751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:10.939124 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:11.071085 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:11.070993 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7"] Oct 30 16:47:11.071496 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:11.071467 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" podUID="8ab63f0b-0a42-4361-ad08-58464c7e65ab" containerName="istio-proxy" containerID="cri-o://b547b5db367875176735f3983c209c5ff8f49489e4a7ef231fe0b81e6794f8f0" gracePeriod=30 Oct 30 16:47:11.124989 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:11.124958 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:11.750679 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:11.750643 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh"] Oct 30 16:47:11.751295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:11.750938 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" podUID="9f783738-9f60-476b-a06c-90f25eda1d4f" containerName="maas-api" containerID="cri-o://7c52f7b67e036af42b517477212a1181c0fce491e283914659bd83726f040a53" gracePeriod=30 Oct 30 16:47:11.841532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:11.841501 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:11.955699 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:11.955663 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn"] Oct 30 16:47:11.955930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:11.955904 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" podUID="af2d6d0c-3ced-4d69-bfc5-732bcf199453" containerName="payload-pre-processing" containerID="cri-o://70637b22a082da0f38c78b2fcbac691fb7493796846865b8faabdf4a96e381a5" gracePeriod=30 Oct 30 16:47:12.069130 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.069099 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:12.092068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.092041 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:47:12.108803 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.108286 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f783738-9f60-476b-a06c-90f25eda1d4f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f783738-9f60-476b-a06c-90f25eda1d4f-maas-api-tls\") pod \"9f783738-9f60-476b-a06c-90f25eda1d4f\" (UID: \"9f783738-9f60-476b-a06c-90f25eda1d4f\") " Oct 30 16:47:12.108803 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.108340 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f783738-9f60-476b-a06c-90f25eda1d4f-kube-api-access-4d78l\" (UniqueName: \"kubernetes.io/projected/9f783738-9f60-476b-a06c-90f25eda1d4f-kube-api-access-4d78l\") pod \"9f783738-9f60-476b-a06c-90f25eda1d4f\" (UID: \"9f783738-9f60-476b-a06c-90f25eda1d4f\") " Oct 30 16:47:12.108803 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.108511 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f783738-9f60-476b-a06c-90f25eda1d4f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f783738-9f60-476b-a06c-90f25eda1d4f-metrics-certs\") pod \"9f783738-9f60-476b-a06c-90f25eda1d4f\" (UID: \"9f783738-9f60-476b-a06c-90f25eda1d4f\") " Oct 30 16:47:12.108803 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.108557 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f783738-9f60-476b-a06c-90f25eda1d4f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f783738-9f60-476b-a06c-90f25eda1d4f-openshift-service-ca\") pod \"9f783738-9f60-476b-a06c-90f25eda1d4f\" (UID: \"9f783738-9f60-476b-a06c-90f25eda1d4f\") " Oct 30 16:47:12.112043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.109334 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f783738-9f60-476b-a06c-90f25eda1d4f-openshift-service-ca" pod "9f783738-9f60-476b-a06c-90f25eda1d4f" (UID: "9f783738-9f60-476b-a06c-90f25eda1d4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:12.112043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.111841 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f783738-9f60-476b-a06c-90f25eda1d4f-maas-api-tls" pod "9f783738-9f60-476b-a06c-90f25eda1d4f" (UID: "9f783738-9f60-476b-a06c-90f25eda1d4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:12.112043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.111882 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f783738-9f60-476b-a06c-90f25eda1d4f-kube-api-access-4d78l" pod "9f783738-9f60-476b-a06c-90f25eda1d4f" (UID: "9f783738-9f60-476b-a06c-90f25eda1d4f"). InnerVolumeSpecName "kube-api-access-4d78l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:12.112577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.112555 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f783738-9f60-476b-a06c-90f25eda1d4f-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:12.112630 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.112585 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f783738-9f60-476b-a06c-90f25eda1d4f-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:12.112630 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.112600 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d78l\" (UniqueName: \"kubernetes.io/projected/9f783738-9f60-476b-a06c-90f25eda1d4f-kube-api-access-4d78l\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:12.113638 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.113618 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f783738-9f60-476b-a06c-90f25eda1d4f-metrics-certs" pod "9f783738-9f60-476b-a06c-90f25eda1d4f" (UID: "9f783738-9f60-476b-a06c-90f25eda1d4f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:12.212485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.212438 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:12.214145 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.214115 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f783738-9f60-476b-a06c-90f25eda1d4f-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:12.238596 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.238576 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" Oct 30 16:47:12.340458 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.338506 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af2d6d0c-3ced-4d69-bfc5-732bcf199453-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2d6d0c-3ced-4d69-bfc5-732bcf199453-plugins-config-volume\") pod \"af2d6d0c-3ced-4d69-bfc5-732bcf199453\" (UID: \"af2d6d0c-3ced-4d69-bfc5-732bcf199453\") " Oct 30 16:47:12.340458 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.338654 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af2d6d0c-3ced-4d69-bfc5-732bcf199453-kube-api-access-5phhb\" (UniqueName: \"kubernetes.io/projected/af2d6d0c-3ced-4d69-bfc5-732bcf199453-kube-api-access-5phhb\") pod \"af2d6d0c-3ced-4d69-bfc5-732bcf199453\" (UID: \"af2d6d0c-3ced-4d69-bfc5-732bcf199453\") " Oct 30 16:47:12.341244 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.341219 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af2d6d0c-3ced-4d69-bfc5-732bcf199453-plugins-config-volume" pod "af2d6d0c-3ced-4d69-bfc5-732bcf199453" (UID: "af2d6d0c-3ced-4d69-bfc5-732bcf199453"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:12.388646 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.388599 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2d6d0c-3ced-4d69-bfc5-732bcf199453-kube-api-access-5phhb" pod "af2d6d0c-3ced-4d69-bfc5-732bcf199453" (UID: "af2d6d0c-3ced-4d69-bfc5-732bcf199453"). InnerVolumeSpecName "kube-api-access-5phhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:12.441934 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.440950 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2d6d0c-3ced-4d69-bfc5-732bcf199453-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:12.441934 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.440992 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5phhb\" (UniqueName: \"kubernetes.io/projected/af2d6d0c-3ced-4d69-bfc5-732bcf199453-kube-api-access-5phhb\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:12.601762 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.601660 4861 generic.go:356] "Generic (PLEG): container finished" podID="af2d6d0c-3ced-4d69-bfc5-732bcf199453" containerID="70637b22a082da0f38c78b2fcbac691fb7493796846865b8faabdf4a96e381a5" exitCode=0 Oct 30 16:47:12.601947 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.601794 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" Oct 30 16:47:12.601947 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.601807 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" event={"ID":"af2d6d0c-3ced-4d69-bfc5-732bcf199453","Type":"ContainerDied","Data":"70637b22a082da0f38c78b2fcbac691fb7493796846865b8faabdf4a96e381a5"} Oct 30 16:47:12.601947 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.601874 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn" event={"ID":"af2d6d0c-3ced-4d69-bfc5-732bcf199453","Type":"ContainerDied","Data":"98fecab976d246156030dbbbe8d11ad5a00270a06d0b132ec1338504d6ae3ce3"} Oct 30 16:47:12.601947 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.601912 4861 scope.go:122] "RemoveContainer" containerID="70637b22a082da0f38c78b2fcbac691fb7493796846865b8faabdf4a96e381a5" Oct 30 16:47:12.603777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.603710 4861 generic.go:356] "Generic (PLEG): container finished" podID="9f783738-9f60-476b-a06c-90f25eda1d4f" containerID="7c52f7b67e036af42b517477212a1181c0fce491e283914659bd83726f040a53" exitCode=0 Oct 30 16:47:12.603918 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.603879 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" event={"ID":"9f783738-9f60-476b-a06c-90f25eda1d4f","Type":"ContainerDied","Data":"7c52f7b67e036af42b517477212a1181c0fce491e283914659bd83726f040a53"} Oct 30 16:47:12.603983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.603922 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" event={"ID":"9f783738-9f60-476b-a06c-90f25eda1d4f","Type":"ContainerDied","Data":"1206e83ba51d48c32c306909a63787e40cc44d2255cd9a806d63947d41d98c0b"} Oct 30 16:47:12.604047 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.603995 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh" Oct 30 16:47:12.615926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.615909 4861 scope.go:122] "RemoveContainer" containerID="70637b22a082da0f38c78b2fcbac691fb7493796846865b8faabdf4a96e381a5" Oct 30 16:47:12.616323 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:12.616298 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70637b22a082da0f38c78b2fcbac691fb7493796846865b8faabdf4a96e381a5\": container with ID starting with 70637b22a082da0f38c78b2fcbac691fb7493796846865b8faabdf4a96e381a5 not found: ID does not exist" containerID="70637b22a082da0f38c78b2fcbac691fb7493796846865b8faabdf4a96e381a5" Oct 30 16:47:12.616461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.616334 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70637b22a082da0f38c78b2fcbac691fb7493796846865b8faabdf4a96e381a5"} err="failed to get container status \"70637b22a082da0f38c78b2fcbac691fb7493796846865b8faabdf4a96e381a5\": rpc error: code = NotFound desc = could not find container \"70637b22a082da0f38c78b2fcbac691fb7493796846865b8faabdf4a96e381a5\": container with ID starting with 70637b22a082da0f38c78b2fcbac691fb7493796846865b8faabdf4a96e381a5 not found: ID does not exist" Oct 30 16:47:12.616461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.616358 4861 scope.go:122] "RemoveContainer" containerID="7c52f7b67e036af42b517477212a1181c0fce491e283914659bd83726f040a53" Oct 30 16:47:12.622150 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.622122 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:12.630545 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.630525 4861 scope.go:122] "RemoveContainer" containerID="7c52f7b67e036af42b517477212a1181c0fce491e283914659bd83726f040a53" Oct 30 16:47:12.630889 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:12.630864 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c52f7b67e036af42b517477212a1181c0fce491e283914659bd83726f040a53\": container with ID starting with 7c52f7b67e036af42b517477212a1181c0fce491e283914659bd83726f040a53 not found: ID does not exist" containerID="7c52f7b67e036af42b517477212a1181c0fce491e283914659bd83726f040a53" Oct 30 16:47:12.631019 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.630893 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c52f7b67e036af42b517477212a1181c0fce491e283914659bd83726f040a53"} err="failed to get container status \"7c52f7b67e036af42b517477212a1181c0fce491e283914659bd83726f040a53\": rpc error: code = NotFound desc = could not find container \"7c52f7b67e036af42b517477212a1181c0fce491e283914659bd83726f040a53\": container with ID starting with 7c52f7b67e036af42b517477212a1181c0fce491e283914659bd83726f040a53 not found: ID does not exist" Oct 30 16:47:12.657987 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.657966 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn"] Oct 30 16:47:12.661118 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.661096 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-4a4c68-789dcc9b95-xzrsn"] Oct 30 16:47:12.711784 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.711761 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh"] Oct 30 16:47:12.717611 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.717588 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4a4c68-598bd546cc-sg2mh"] Oct 30 16:47:12.795195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.791795 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:12.944894 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:12.944028 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:13.263376 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:13.263340 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:13.808102 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:13.808071 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f783738-9f60-476b-a06c-90f25eda1d4f" path="/var/lib/kubelet/pods/9f783738-9f60-476b-a06c-90f25eda1d4f/volumes" Oct 30 16:47:13.808489 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:13.808475 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af2d6d0c-3ced-4d69-bfc5-732bcf199453" path="/var/lib/kubelet/pods/af2d6d0c-3ced-4d69-bfc5-732bcf199453/volumes" Oct 30 16:47:14.430906 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.430870 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:14.646826 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.646797 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.990528 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz"] Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.992428 4861 cpu_manager.go:414] "containerMap: removing container" podUID="e88f463a-7f54-49ba-94da-f6522fe52aa2" containerName="maas-api" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.992459 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e88f463a-7f54-49ba-94da-f6522fe52aa2" containerName="maas-api" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.992478 4861 cpu_manager.go:414] "containerMap: removing container" podUID="2e07cfe3-baff-49e7-8bf7-73715fa20049" containerName="payload-pre-processing" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.992486 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e07cfe3-baff-49e7-8bf7-73715fa20049" containerName="payload-pre-processing" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.992531 4861 cpu_manager.go:414] "containerMap: removing container" podUID="5e75dc21-662f-4048-8455-df78d3db3b9a" containerName="payload-pre-processing" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.992539 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e75dc21-662f-4048-8455-df78d3db3b9a" containerName="payload-pre-processing" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.992555 4861 cpu_manager.go:414] "containerMap: removing container" podUID="9f783738-9f60-476b-a06c-90f25eda1d4f" containerName="maas-api" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.992563 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f783738-9f60-476b-a06c-90f25eda1d4f" containerName="maas-api" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.992596 4861 cpu_manager.go:414] "containerMap: removing container" podUID="6ecf9d2f-4389-46e8-801d-970f54fe8a14" containerName="payload-processing" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.992603 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ecf9d2f-4389-46e8-801d-970f54fe8a14" containerName="payload-processing" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.992623 4861 cpu_manager.go:414] "containerMap: removing container" podUID="45769419-34ec-42d9-b5de-035877d01d2b" containerName="maas-api" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.992630 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45769419-34ec-42d9-b5de-035877d01d2b" containerName="maas-api" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.992782 4861 cpu_manager.go:414] "containerMap: removing container" podUID="af2d6d0c-3ced-4d69-bfc5-732bcf199453" containerName="payload-pre-processing" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.992796 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af2d6d0c-3ced-4d69-bfc5-732bcf199453" containerName="payload-pre-processing" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.993173 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e75dc21-662f-4048-8455-df78d3db3b9a" containerName="payload-pre-processing" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.993201 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f783738-9f60-476b-a06c-90f25eda1d4f" containerName="maas-api" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.993223 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ecf9d2f-4389-46e8-801d-970f54fe8a14" containerName="payload-processing" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.993241 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="45769419-34ec-42d9-b5de-035877d01d2b" containerName="maas-api" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.993250 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="af2d6d0c-3ced-4d69-bfc5-732bcf199453" containerName="payload-pre-processing" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.993273 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="e88f463a-7f54-49ba-94da-f6522fe52aa2" containerName="maas-api" Oct 30 16:47:14.994835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:14.993286 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e07cfe3-baff-49e7-8bf7-73715fa20049" containerName="payload-pre-processing" Oct 30 16:47:15.020788 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.018686 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" Oct 30 16:47:15.036917 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.036888 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-06ac18\"" Oct 30 16:47:15.043036 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.042804 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz"] Oct 30 16:47:15.126988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.126959 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1916010e-81c5-4927-af79-d4d9b343ace4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz\" (UID: \"1916010e-81c5-4927-af79-d4d9b343ace4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" Oct 30 16:47:15.127209 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.127191 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7phvm\" (UniqueName: \"kubernetes.io/projected/1916010e-81c5-4927-af79-d4d9b343ace4-kube-api-access-7phvm\") pod \"payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz\" (UID: \"1916010e-81c5-4927-af79-d4d9b343ace4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" Oct 30 16:47:15.133127 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.133106 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-06ac18-dockercfg-hwggk\"" Oct 30 16:47:15.151628 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.151603 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz"] Oct 30 16:47:15.156247 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.156227 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" Oct 30 16:47:15.178850 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.178831 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz"] Oct 30 16:47:15.228916 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.228888 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1916010e-81c5-4927-af79-d4d9b343ace4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz\" (UID: \"1916010e-81c5-4927-af79-d4d9b343ace4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" Oct 30 16:47:15.229117 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.229097 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7phvm\" (UniqueName: \"kubernetes.io/projected/1916010e-81c5-4927-af79-d4d9b343ace4-kube-api-access-7phvm\") pod \"payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz\" (UID: \"1916010e-81c5-4927-af79-d4d9b343ace4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" Oct 30 16:47:15.229623 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.229605 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1916010e-81c5-4927-af79-d4d9b343ace4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz\" (UID: \"1916010e-81c5-4927-af79-d4d9b343ace4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" Oct 30 16:47:15.252846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.252798 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7phvm\" (UniqueName: \"kubernetes.io/projected/1916010e-81c5-4927-af79-d4d9b343ace4-kube-api-access-7phvm\") pod \"payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz\" (UID: \"1916010e-81c5-4927-af79-d4d9b343ace4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" Oct 30 16:47:15.330876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.330839 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7rr6\" (UniqueName: \"kubernetes.io/projected/d3e15e44-d162-4dc6-9c52-8ac1be42411f-kube-api-access-s7rr6\") pod \"payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz\" (UID: \"d3e15e44-d162-4dc6-9c52-8ac1be42411f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" Oct 30 16:47:15.331078 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.331056 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e15e44-d162-4dc6-9c52-8ac1be42411f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz\" (UID: \"d3e15e44-d162-4dc6-9c52-8ac1be42411f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" Oct 30 16:47:15.347940 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.347919 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" Oct 30 16:47:15.439785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.436342 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e15e44-d162-4dc6-9c52-8ac1be42411f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz\" (UID: \"d3e15e44-d162-4dc6-9c52-8ac1be42411f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" Oct 30 16:47:15.439785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.436529 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7rr6\" (UniqueName: \"kubernetes.io/projected/d3e15e44-d162-4dc6-9c52-8ac1be42411f-kube-api-access-s7rr6\") pod \"payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz\" (UID: \"d3e15e44-d162-4dc6-9c52-8ac1be42411f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" Oct 30 16:47:15.439785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.437842 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e15e44-d162-4dc6-9c52-8ac1be42411f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz\" (UID: \"d3e15e44-d162-4dc6-9c52-8ac1be42411f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" Oct 30 16:47:15.461867 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.461835 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7rr6\" (UniqueName: \"kubernetes.io/projected/d3e15e44-d162-4dc6-9c52-8ac1be42411f-kube-api-access-s7rr6\") pod \"payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz\" (UID: \"d3e15e44-d162-4dc6-9c52-8ac1be42411f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" Oct 30 16:47:15.470505 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.470483 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" Oct 30 16:47:15.667819 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.667795 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz"] Oct 30 16:47:15.668813 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:47:15.668788 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1916010e_81c5_4927_af79_d4d9b343ace4.slice/crio-1c0c368bb39f0044f427f6207790e17f652f070f1151eae3fdee36607c290f68 WatchSource:0}: Error finding container 1c0c368bb39f0044f427f6207790e17f652f070f1151eae3fdee36607c290f68: Status 404 returned error can't find the container with id 1c0c368bb39f0044f427f6207790e17f652f070f1151eae3fdee36607c290f68 Oct 30 16:47:15.796032 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.796004 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz"] Oct 30 16:47:15.796866 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:47:15.796840 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e15e44_d162_4dc6_9c52_8ac1be42411f.slice/crio-b34a55bce64f876303a75671470f70646ba66d1c4f2d89eeed74d27cc83ea210 WatchSource:0}: Error finding container b34a55bce64f876303a75671470f70646ba66d1c4f2d89eeed74d27cc83ea210: Status 404 returned error can't find the container with id b34a55bce64f876303a75671470f70646ba66d1c4f2d89eeed74d27cc83ea210 Oct 30 16:47:15.887460 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:15.887423 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:16.369298 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.369274 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:47:16.451257 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.451221 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab63f0b-0a42-4361-ad08-58464c7e65ab-kube-api-access-m59ws\" (UniqueName: \"kubernetes.io/projected/8ab63f0b-0a42-4361-ad08-58464c7e65ab-kube-api-access-m59ws\") pod \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " Oct 30 16:47:16.451471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.451400 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-podinfo\") pod \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " Oct 30 16:47:16.451471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.451443 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-envoy\") pod \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " Oct 30 16:47:16.451605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.451527 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-token\" (UniqueName: \"kubernetes.io/projected/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-token\") pod \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " Oct 30 16:47:16.451605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.451589 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istiod-ca-cert\") pod \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " Oct 30 16:47:16.451763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.451634 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-data\") pod \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " Oct 30 16:47:16.451763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.451683 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-workload-socket\") pod \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " Oct 30 16:47:16.451763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.451713 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-workload-certs\") pod \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " Oct 30 16:47:16.451957 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.451787 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-credential-socket\") pod \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\" (UID: \"8ab63f0b-0a42-4361-ad08-58464c7e65ab\") " Oct 30 16:47:16.452135 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.452100 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istiod-ca-cert" pod "8ab63f0b-0a42-4361-ad08-58464c7e65ab" (UID: "8ab63f0b-0a42-4361-ad08-58464c7e65ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:16.452398 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.452377 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-data" pod "8ab63f0b-0a42-4361-ad08-58464c7e65ab" (UID: "8ab63f0b-0a42-4361-ad08-58464c7e65ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:16.452522 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.452473 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-workload-certs" pod "8ab63f0b-0a42-4361-ad08-58464c7e65ab" (UID: "8ab63f0b-0a42-4361-ad08-58464c7e65ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:16.452582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.452563 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-workload-socket" pod "8ab63f0b-0a42-4361-ad08-58464c7e65ab" (UID: "8ab63f0b-0a42-4361-ad08-58464c7e65ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:16.452681 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.452661 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-credential-socket" pod "8ab63f0b-0a42-4361-ad08-58464c7e65ab" (UID: "8ab63f0b-0a42-4361-ad08-58464c7e65ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:16.453380 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.453355 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:16.453531 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.453384 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:16.453531 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.453398 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:16.453531 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.453411 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:16.453531 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.453424 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:16.454577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.454542 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-podinfo" pod "8ab63f0b-0a42-4361-ad08-58464c7e65ab" (UID: "8ab63f0b-0a42-4361-ad08-58464c7e65ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:47:16.456589 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.456565 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-envoy" pod "8ab63f0b-0a42-4361-ad08-58464c7e65ab" (UID: "8ab63f0b-0a42-4361-ad08-58464c7e65ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:16.456875 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.456841 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab63f0b-0a42-4361-ad08-58464c7e65ab-kube-api-access-m59ws" pod "8ab63f0b-0a42-4361-ad08-58464c7e65ab" (UID: "8ab63f0b-0a42-4361-ad08-58464c7e65ab"). InnerVolumeSpecName "kube-api-access-m59ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:16.456961 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.456903 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-token" pod "8ab63f0b-0a42-4361-ad08-58464c7e65ab" (UID: "8ab63f0b-0a42-4361-ad08-58464c7e65ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:16.554242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.554216 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m59ws\" (UniqueName: \"kubernetes.io/projected/8ab63f0b-0a42-4361-ad08-58464c7e65ab-kube-api-access-m59ws\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:16.554242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.554240 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:16.554452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.554250 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:16.554452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.554258 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ab63f0b-0a42-4361-ad08-58464c7e65ab-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:16.668350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.668268 4861 generic.go:356] "Generic (PLEG): container finished" podID="8ab63f0b-0a42-4361-ad08-58464c7e65ab" containerID="b547b5db367875176735f3983c209c5ff8f49489e4a7ef231fe0b81e6794f8f0" exitCode=0 Oct 30 16:47:16.668350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.668316 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" Oct 30 16:47:16.668549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.668349 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" event={"ID":"8ab63f0b-0a42-4361-ad08-58464c7e65ab","Type":"ContainerDied","Data":"b547b5db367875176735f3983c209c5ff8f49489e4a7ef231fe0b81e6794f8f0"} Oct 30 16:47:16.668549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.668400 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7" event={"ID":"8ab63f0b-0a42-4361-ad08-58464c7e65ab","Type":"ContainerDied","Data":"0f5d50a64e5be1d5b561ee484c3f659fc7d12e1bffb5b9348e203b8f07fc2c56"} Oct 30 16:47:16.668549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.668430 4861 scope.go:122] "RemoveContainer" containerID="b547b5db367875176735f3983c209c5ff8f49489e4a7ef231fe0b81e6794f8f0" Oct 30 16:47:16.670402 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.670374 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" event={"ID":"d3e15e44-d162-4dc6-9c52-8ac1be42411f","Type":"ContainerStarted","Data":"6b6d163e16d031dd34af88c7648301409573ea3b3d6482f8f6a04ecb34696f65"} Oct 30 16:47:16.670536 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.670436 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" Oct 30 16:47:16.670536 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.670463 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" event={"ID":"d3e15e44-d162-4dc6-9c52-8ac1be42411f","Type":"ContainerStarted","Data":"b34a55bce64f876303a75671470f70646ba66d1c4f2d89eeed74d27cc83ea210"} Oct 30 16:47:16.672321 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.672300 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" event={"ID":"1916010e-81c5-4927-af79-d4d9b343ace4","Type":"ContainerStarted","Data":"bb8ac86aa8771f758ed4b7b8ed8b98045642f933fae2c33bd34eca5114822b17"} Oct 30 16:47:16.672426 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.672338 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" event={"ID":"1916010e-81c5-4927-af79-d4d9b343ace4","Type":"ContainerStarted","Data":"1c0c368bb39f0044f427f6207790e17f652f070f1151eae3fdee36607c290f68"} Oct 30 16:47:16.672493 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.672439 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" Oct 30 16:47:16.680468 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.680444 4861 scope.go:122] "RemoveContainer" containerID="2a9faf03ba898ac4ce5e3f6d54207081d5c965ad22207174e7bb726cd70eeaef" Oct 30 16:47:16.690167 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.690147 4861 scope.go:122] "RemoveContainer" containerID="b547b5db367875176735f3983c209c5ff8f49489e4a7ef231fe0b81e6794f8f0" Oct 30 16:47:16.690485 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:16.690464 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b547b5db367875176735f3983c209c5ff8f49489e4a7ef231fe0b81e6794f8f0\": container with ID starting with b547b5db367875176735f3983c209c5ff8f49489e4a7ef231fe0b81e6794f8f0 not found: ID does not exist" containerID="b547b5db367875176735f3983c209c5ff8f49489e4a7ef231fe0b81e6794f8f0" Oct 30 16:47:16.690582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.690493 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b547b5db367875176735f3983c209c5ff8f49489e4a7ef231fe0b81e6794f8f0"} err="failed to get container status \"b547b5db367875176735f3983c209c5ff8f49489e4a7ef231fe0b81e6794f8f0\": rpc error: code = NotFound desc = could not find container \"b547b5db367875176735f3983c209c5ff8f49489e4a7ef231fe0b81e6794f8f0\": container with ID starting with b547b5db367875176735f3983c209c5ff8f49489e4a7ef231fe0b81e6794f8f0 not found: ID does not exist" Oct 30 16:47:16.690582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.690519 4861 scope.go:122] "RemoveContainer" containerID="2a9faf03ba898ac4ce5e3f6d54207081d5c965ad22207174e7bb726cd70eeaef" Oct 30 16:47:16.691269 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:16.691242 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a9faf03ba898ac4ce5e3f6d54207081d5c965ad22207174e7bb726cd70eeaef\": container with ID starting with 2a9faf03ba898ac4ce5e3f6d54207081d5c965ad22207174e7bb726cd70eeaef not found: ID does not exist" containerID="2a9faf03ba898ac4ce5e3f6d54207081d5c965ad22207174e7bb726cd70eeaef" Oct 30 16:47:16.691342 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.691279 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9faf03ba898ac4ce5e3f6d54207081d5c965ad22207174e7bb726cd70eeaef"} err="failed to get container status \"2a9faf03ba898ac4ce5e3f6d54207081d5c965ad22207174e7bb726cd70eeaef\": rpc error: code = NotFound desc = could not find container \"2a9faf03ba898ac4ce5e3f6d54207081d5c965ad22207174e7bb726cd70eeaef\": container with ID starting with 2a9faf03ba898ac4ce5e3f6d54207081d5c965ad22207174e7bb726cd70eeaef not found: ID does not exist" Oct 30 16:47:16.723331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.723305 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7"] Oct 30 16:47:16.735078 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.735056 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-843ab3-openshift-default-7bfd5989d5-l72f7"] Oct 30 16:47:16.774588 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.774549 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" podStartSLOduration=2.774537671 podStartE2EDuration="2.774537671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:16.772547247 +0000 UTC m=+1467.115221289" watchObservedRunningTime="2026-10-30 16:47:16.774537671 +0000 UTC m=+1467.117211702" Oct 30 16:47:16.816122 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:16.816073 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" podStartSLOduration=1.8160581329999999 podStartE2EDuration="1.816058133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:16.813393822 +0000 UTC m=+1467.156067864" watchObservedRunningTime="2026-10-30 16:47:16.816058133 +0000 UTC m=+1467.158732174" Oct 30 16:47:17.407689 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.407658 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q"] Oct 30 16:47:17.408144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.407898 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" podUID="8112653e-e8ad-4214-add5-acf873438708" containerName="maas-api" containerID="cri-o://a35fc09da9a0feb39458a3db8be1780966592052f35094c2249d52fb6af08547" gracePeriod=30 Oct 30 16:47:17.500128 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.500074 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:17.573357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.573324 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8"] Oct 30 16:47:17.573585 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.573560 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" podUID="4ba88b21-2400-499d-960d-03320e5b9dcd" containerName="payload-processing" containerID="cri-o://a7b8216c50fa63a48a98ecaff3a2e903447f2bc5fee4d56d160b2a3e7d9d7a04" gracePeriod=30 Oct 30 16:47:17.629991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.628888 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9"] Oct 30 16:47:17.629991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.629111 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" podUID="df9fefa6-cb76-453e-b5c2-37e429f2cd5d" containerName="payload-pre-processing" containerID="cri-o://8ab51314ed849e8081bf494857995635f56f437d6630d13597608716dcce90e4" gracePeriod=30 Oct 30 16:47:17.695065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.695018 4861 generic.go:356] "Generic (PLEG): container finished" podID="8112653e-e8ad-4214-add5-acf873438708" containerID="a35fc09da9a0feb39458a3db8be1780966592052f35094c2249d52fb6af08547" exitCode=0 Oct 30 16:47:17.695233 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.695082 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" event={"ID":"8112653e-e8ad-4214-add5-acf873438708","Type":"ContainerDied","Data":"a35fc09da9a0feb39458a3db8be1780966592052f35094c2249d52fb6af08547"} Oct 30 16:47:17.696538 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.696508 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:17.769409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.769379 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:47:17.815746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.810679 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab63f0b-0a42-4361-ad08-58464c7e65ab" path="/var/lib/kubelet/pods/8ab63f0b-0a42-4361-ad08-58464c7e65ab/volumes" Oct 30 16:47:17.848827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.844993 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:17.874414 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.874380 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8112653e-e8ad-4214-add5-acf873438708-kube-api-access-nmhjn\" (UniqueName: \"kubernetes.io/projected/8112653e-e8ad-4214-add5-acf873438708-kube-api-access-nmhjn\") pod \"8112653e-e8ad-4214-add5-acf873438708\" (UID: \"8112653e-e8ad-4214-add5-acf873438708\") " Oct 30 16:47:17.874658 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.874627 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8112653e-e8ad-4214-add5-acf873438708-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8112653e-e8ad-4214-add5-acf873438708-openshift-service-ca\") pod \"8112653e-e8ad-4214-add5-acf873438708\" (UID: \"8112653e-e8ad-4214-add5-acf873438708\") " Oct 30 16:47:17.874782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.874677 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8112653e-e8ad-4214-add5-acf873438708-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8112653e-e8ad-4214-add5-acf873438708-maas-api-tls\") pod \"8112653e-e8ad-4214-add5-acf873438708\" (UID: \"8112653e-e8ad-4214-add5-acf873438708\") " Oct 30 16:47:17.874862 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.874824 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8112653e-e8ad-4214-add5-acf873438708-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8112653e-e8ad-4214-add5-acf873438708-metrics-certs\") pod \"8112653e-e8ad-4214-add5-acf873438708\" (UID: \"8112653e-e8ad-4214-add5-acf873438708\") " Oct 30 16:47:17.875402 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.875221 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8112653e-e8ad-4214-add5-acf873438708-openshift-service-ca" pod "8112653e-e8ad-4214-add5-acf873438708" (UID: "8112653e-e8ad-4214-add5-acf873438708"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:17.879854 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.878330 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8112653e-e8ad-4214-add5-acf873438708-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:17.880903 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.880875 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8112653e-e8ad-4214-add5-acf873438708-metrics-certs" pod "8112653e-e8ad-4214-add5-acf873438708" (UID: "8112653e-e8ad-4214-add5-acf873438708"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:17.881327 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.881287 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8112653e-e8ad-4214-add5-acf873438708-maas-api-tls" pod "8112653e-e8ad-4214-add5-acf873438708" (UID: "8112653e-e8ad-4214-add5-acf873438708"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:17.884666 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:17.884645 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8112653e-e8ad-4214-add5-acf873438708-kube-api-access-nmhjn" pod "8112653e-e8ad-4214-add5-acf873438708" (UID: "8112653e-e8ad-4214-add5-acf873438708"). InnerVolumeSpecName "kube-api-access-nmhjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:18.123750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.115080 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmhjn\" (UniqueName: \"kubernetes.io/projected/8112653e-e8ad-4214-add5-acf873438708-kube-api-access-nmhjn\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:18.123750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.115151 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8112653e-e8ad-4214-add5-acf873438708-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:18.123750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.115167 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8112653e-e8ad-4214-add5-acf873438708-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:18.180857 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.153551 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:18.194683 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.194646 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" Oct 30 16:47:18.216224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.216193 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba88b21-2400-499d-960d-03320e5b9dcd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba88b21-2400-499d-960d-03320e5b9dcd-plugins-config-volume\") pod \"4ba88b21-2400-499d-960d-03320e5b9dcd\" (UID: \"4ba88b21-2400-499d-960d-03320e5b9dcd\") " Oct 30 16:47:18.216348 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.216319 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba88b21-2400-499d-960d-03320e5b9dcd-kube-api-access-fr6fc\" (UniqueName: \"kubernetes.io/projected/4ba88b21-2400-499d-960d-03320e5b9dcd-kube-api-access-fr6fc\") pod \"4ba88b21-2400-499d-960d-03320e5b9dcd\" (UID: \"4ba88b21-2400-499d-960d-03320e5b9dcd\") " Oct 30 16:47:18.216562 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.216542 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba88b21-2400-499d-960d-03320e5b9dcd-plugins-config-volume" pod "4ba88b21-2400-499d-960d-03320e5b9dcd" (UID: "4ba88b21-2400-499d-960d-03320e5b9dcd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:18.216688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.216673 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba88b21-2400-499d-960d-03320e5b9dcd-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:18.218648 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.218628 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba88b21-2400-499d-960d-03320e5b9dcd-kube-api-access-fr6fc" pod "4ba88b21-2400-499d-960d-03320e5b9dcd" (UID: "4ba88b21-2400-499d-960d-03320e5b9dcd"). InnerVolumeSpecName "kube-api-access-fr6fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:18.220111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.220094 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" Oct 30 16:47:18.307410 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.307377 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:18.317420 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.317391 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df9fefa6-cb76-453e-b5c2-37e429f2cd5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9fefa6-cb76-453e-b5c2-37e429f2cd5d-plugins-config-volume\") pod \"df9fefa6-cb76-453e-b5c2-37e429f2cd5d\" (UID: \"df9fefa6-cb76-453e-b5c2-37e429f2cd5d\") " Oct 30 16:47:18.317657 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.317635 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df9fefa6-cb76-453e-b5c2-37e429f2cd5d-kube-api-access-7zq2w\" (UniqueName: \"kubernetes.io/projected/df9fefa6-cb76-453e-b5c2-37e429f2cd5d-kube-api-access-7zq2w\") pod \"df9fefa6-cb76-453e-b5c2-37e429f2cd5d\" (UID: \"df9fefa6-cb76-453e-b5c2-37e429f2cd5d\") " Oct 30 16:47:18.318264 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.318237 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr6fc\" (UniqueName: \"kubernetes.io/projected/4ba88b21-2400-499d-960d-03320e5b9dcd-kube-api-access-fr6fc\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:18.320894 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.320864 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df9fefa6-cb76-453e-b5c2-37e429f2cd5d-kube-api-access-7zq2w" pod "df9fefa6-cb76-453e-b5c2-37e429f2cd5d" (UID: "df9fefa6-cb76-453e-b5c2-37e429f2cd5d"). InnerVolumeSpecName "kube-api-access-7zq2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:18.322755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.322707 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df9fefa6-cb76-453e-b5c2-37e429f2cd5d-plugins-config-volume" pod "df9fefa6-cb76-453e-b5c2-37e429f2cd5d" (UID: "df9fefa6-cb76-453e-b5c2-37e429f2cd5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:18.433799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.426316 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zq2w\" (UniqueName: \"kubernetes.io/projected/df9fefa6-cb76-453e-b5c2-37e429f2cd5d-kube-api-access-7zq2w\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:18.433799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.426374 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df9fefa6-cb76-453e-b5c2-37e429f2cd5d-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:18.457431 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.457299 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:18.704411 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.704328 4861 generic.go:356] "Generic (PLEG): container finished" podID="df9fefa6-cb76-453e-b5c2-37e429f2cd5d" containerID="8ab51314ed849e8081bf494857995635f56f437d6630d13597608716dcce90e4" exitCode=0 Oct 30 16:47:18.704411 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.704383 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" Oct 30 16:47:18.704628 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.704401 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" event={"ID":"df9fefa6-cb76-453e-b5c2-37e429f2cd5d","Type":"ContainerDied","Data":"8ab51314ed849e8081bf494857995635f56f437d6630d13597608716dcce90e4"} Oct 30 16:47:18.704628 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.704469 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9" event={"ID":"df9fefa6-cb76-453e-b5c2-37e429f2cd5d","Type":"ContainerDied","Data":"10073a912a06332d48f7a42e60ea511089c8544448fb14933547bd8ea4e40ebf"} Oct 30 16:47:18.704628 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.704502 4861 scope.go:122] "RemoveContainer" containerID="8ab51314ed849e8081bf494857995635f56f437d6630d13597608716dcce90e4" Oct 30 16:47:18.706113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.706080 4861 generic.go:356] "Generic (PLEG): container finished" podID="4ba88b21-2400-499d-960d-03320e5b9dcd" containerID="a7b8216c50fa63a48a98ecaff3a2e903447f2bc5fee4d56d160b2a3e7d9d7a04" exitCode=0 Oct 30 16:47:18.706236 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.706141 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" Oct 30 16:47:18.706236 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.706188 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" event={"ID":"4ba88b21-2400-499d-960d-03320e5b9dcd","Type":"ContainerDied","Data":"a7b8216c50fa63a48a98ecaff3a2e903447f2bc5fee4d56d160b2a3e7d9d7a04"} Oct 30 16:47:18.706324 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.706235 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8" event={"ID":"4ba88b21-2400-499d-960d-03320e5b9dcd","Type":"ContainerDied","Data":"5ed0b9f70222ed20ef0a09d03cc1863b599010cd00c6d901fb70def3c3a41d7c"} Oct 30 16:47:18.708159 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.708139 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" Oct 30 16:47:18.708296 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.708151 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q" event={"ID":"8112653e-e8ad-4214-add5-acf873438708","Type":"ContainerDied","Data":"b55a59d173ecc5ea45691266ee34437acaec14c4749357364b8119a3a5874d06"} Oct 30 16:47:18.717341 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.717323 4861 scope.go:122] "RemoveContainer" containerID="8ab51314ed849e8081bf494857995635f56f437d6630d13597608716dcce90e4" Oct 30 16:47:18.717642 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:18.717612 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab51314ed849e8081bf494857995635f56f437d6630d13597608716dcce90e4\": container with ID starting with 8ab51314ed849e8081bf494857995635f56f437d6630d13597608716dcce90e4 not found: ID does not exist" containerID="8ab51314ed849e8081bf494857995635f56f437d6630d13597608716dcce90e4" Oct 30 16:47:18.717772 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.717650 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab51314ed849e8081bf494857995635f56f437d6630d13597608716dcce90e4"} err="failed to get container status \"8ab51314ed849e8081bf494857995635f56f437d6630d13597608716dcce90e4\": rpc error: code = NotFound desc = could not find container \"8ab51314ed849e8081bf494857995635f56f437d6630d13597608716dcce90e4\": container with ID starting with 8ab51314ed849e8081bf494857995635f56f437d6630d13597608716dcce90e4 not found: ID does not exist" Oct 30 16:47:18.717772 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.717673 4861 scope.go:122] "RemoveContainer" containerID="a7b8216c50fa63a48a98ecaff3a2e903447f2bc5fee4d56d160b2a3e7d9d7a04" Oct 30 16:47:18.727981 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.727965 4861 scope.go:122] "RemoveContainer" containerID="a7b8216c50fa63a48a98ecaff3a2e903447f2bc5fee4d56d160b2a3e7d9d7a04" Oct 30 16:47:18.728240 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:18.728223 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b8216c50fa63a48a98ecaff3a2e903447f2bc5fee4d56d160b2a3e7d9d7a04\": container with ID starting with a7b8216c50fa63a48a98ecaff3a2e903447f2bc5fee4d56d160b2a3e7d9d7a04 not found: ID does not exist" containerID="a7b8216c50fa63a48a98ecaff3a2e903447f2bc5fee4d56d160b2a3e7d9d7a04" Oct 30 16:47:18.728325 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.728248 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b8216c50fa63a48a98ecaff3a2e903447f2bc5fee4d56d160b2a3e7d9d7a04"} err="failed to get container status \"a7b8216c50fa63a48a98ecaff3a2e903447f2bc5fee4d56d160b2a3e7d9d7a04\": rpc error: code = NotFound desc = could not find container \"a7b8216c50fa63a48a98ecaff3a2e903447f2bc5fee4d56d160b2a3e7d9d7a04\": container with ID starting with a7b8216c50fa63a48a98ecaff3a2e903447f2bc5fee4d56d160b2a3e7d9d7a04 not found: ID does not exist" Oct 30 16:47:18.728325 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.728270 4861 scope.go:122] "RemoveContainer" containerID="a35fc09da9a0feb39458a3db8be1780966592052f35094c2249d52fb6af08547" Oct 30 16:47:18.770510 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.770488 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:18.792988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.792968 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9"] Oct 30 16:47:18.805217 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.805197 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c23a4-6f77c4447-87zw9"] Oct 30 16:47:18.832426 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.832406 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8"] Oct 30 16:47:18.836342 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.836324 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7c23a4-f78cd9858-xkfh8"] Oct 30 16:47:18.872035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.872007 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q"] Oct 30 16:47:18.875956 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:18.875934 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7c23a4-6d567d54b6-cjn4q"] Oct 30 16:47:19.074969 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:19.074934 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf"] Oct 30 16:47:19.108389 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:19.108363 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v"] Oct 30 16:47:19.252160 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:19.252131 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:19.398582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:19.398508 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:19.862750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:19.858978 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba88b21-2400-499d-960d-03320e5b9dcd" path="/var/lib/kubelet/pods/4ba88b21-2400-499d-960d-03320e5b9dcd/volumes" Oct 30 16:47:19.862750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:19.859641 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8112653e-e8ad-4214-add5-acf873438708" path="/var/lib/kubelet/pods/8112653e-e8ad-4214-add5-acf873438708/volumes" Oct 30 16:47:19.862750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:19.861494 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df9fefa6-cb76-453e-b5c2-37e429f2cd5d" path="/var/lib/kubelet/pods/df9fefa6-cb76-453e-b5c2-37e429f2cd5d/volumes" Oct 30 16:47:20.289065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:20.289028 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:20.290711 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:20.290686 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:47:20.428743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:20.428690 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:20.570988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:20.570897 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h"] Oct 30 16:47:20.571293 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:20.571261 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" podUID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerName="istio-proxy" containerID="cri-o://1b8f9b0624d4be8c381d1cd208566d752a17f871245576d8c365aadfab04fe65" gracePeriod=30 Oct 30 16:47:20.579277 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:20.579230 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:20.732875 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:20.724214 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:20.733164 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:20.728159 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:47:20.733355 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:20.733332 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" podUID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:47:22.155193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.155160 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:22.486004 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.485974 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:22.535483 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.535456 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7"] Oct 30 16:47:22.536450 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.536413 4861 cpu_manager.go:414] "containerMap: removing container" podUID="4ba88b21-2400-499d-960d-03320e5b9dcd" containerName="payload-processing" Oct 30 16:47:22.536450 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.536441 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba88b21-2400-499d-960d-03320e5b9dcd" containerName="payload-processing" Oct 30 16:47:22.536686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.536491 4861 cpu_manager.go:414] "containerMap: removing container" podUID="df9fefa6-cb76-453e-b5c2-37e429f2cd5d" containerName="payload-pre-processing" Oct 30 16:47:22.536686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.536499 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df9fefa6-cb76-453e-b5c2-37e429f2cd5d" containerName="payload-pre-processing" Oct 30 16:47:22.536686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.536536 4861 cpu_manager.go:414] "containerMap: removing container" podUID="8ab63f0b-0a42-4361-ad08-58464c7e65ab" containerName="istio-proxy" Oct 30 16:47:22.536686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.536544 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab63f0b-0a42-4361-ad08-58464c7e65ab" containerName="istio-proxy" Oct 30 16:47:22.536686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.536554 4861 cpu_manager.go:414] "containerMap: removing container" podUID="8ab63f0b-0a42-4361-ad08-58464c7e65ab" containerName="istio-proxy" Oct 30 16:47:22.536686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.536561 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab63f0b-0a42-4361-ad08-58464c7e65ab" containerName="istio-proxy" Oct 30 16:47:22.536686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.536574 4861 cpu_manager.go:414] "containerMap: removing container" podUID="8112653e-e8ad-4214-add5-acf873438708" containerName="maas-api" Oct 30 16:47:22.536686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.536581 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8112653e-e8ad-4214-add5-acf873438708" containerName="maas-api" Oct 30 16:47:22.537320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.536743 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba88b21-2400-499d-960d-03320e5b9dcd" containerName="payload-processing" Oct 30 16:47:22.537320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.536772 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="df9fefa6-cb76-453e-b5c2-37e429f2cd5d" containerName="payload-pre-processing" Oct 30 16:47:22.537320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.536790 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab63f0b-0a42-4361-ad08-58464c7e65ab" containerName="istio-proxy" Oct 30 16:47:22.537320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.536800 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab63f0b-0a42-4361-ad08-58464c7e65ab" containerName="istio-proxy" Oct 30 16:47:22.537320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.536812 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="8112653e-e8ad-4214-add5-acf873438708" containerName="maas-api" Oct 30 16:47:22.542675 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.542654 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.559452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.559313 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-569b62-openshift-default-dockercfg-kkt8c\"" Oct 30 16:47:22.564886 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.564864 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-podinfo\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.565011 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.564917 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-data\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.565082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.565042 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-credential-socket\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.565146 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.565112 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69254306-affa-4f98-a1a0-93c0e29c14ae-istiod-ca-cert\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.565211 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.565195 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltzhl\" (UniqueName: \"kubernetes.io/projected/69254306-affa-4f98-a1a0-93c0e29c14ae-kube-api-access-ltzhl\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.565272 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.565257 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-workload-certs\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.565330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.565290 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-workload-socket\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.565330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.565308 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-token\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.565441 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.565361 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-envoy\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.584836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.584810 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7"] Oct 30 16:47:22.625097 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.625071 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:22.666770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.666716 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-envoy\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.666896 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.666784 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-podinfo\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.666896 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.666841 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-data\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.666896 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.666880 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-credential-socket\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.667029 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.666962 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69254306-affa-4f98-a1a0-93c0e29c14ae-istiod-ca-cert\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.667078 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.667046 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltzhl\" (UniqueName: \"kubernetes.io/projected/69254306-affa-4f98-a1a0-93c0e29c14ae-kube-api-access-ltzhl\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.667335 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.667311 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-credential-socket\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.667428 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.667323 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-data\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.667551 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.667525 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-workload-certs\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.667624 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.667580 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-workload-certs\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.667688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.667654 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-workload-socket\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.667789 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.667685 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-token\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.667980 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.667957 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69254306-affa-4f98-a1a0-93c0e29c14ae-istiod-ca-cert\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.668027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.667977 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-workload-socket\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.669340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.669317 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-envoy\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.669439 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.669376 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-podinfo\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.731756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.725979 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltzhl\" (UniqueName: \"kubernetes.io/projected/69254306-affa-4f98-a1a0-93c0e29c14ae-kube-api-access-ltzhl\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.771598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.771484 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:22.772467 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.772404 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-token\") pod \"e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:22.864747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:22.863127 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:23.281837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:23.281571 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7"] Oct 30 16:47:23.292288 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:47:23.292248 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69254306_affa_4f98_a1a0_93c0e29c14ae.slice/crio-f806052ad1f31206d4ba151c11573d4aa9135e2bdb17ec4e506a7ffc35384f60 WatchSource:0}: Error finding container f806052ad1f31206d4ba151c11573d4aa9135e2bdb17ec4e506a7ffc35384f60: Status 404 returned error can't find the container with id f806052ad1f31206d4ba151c11573d4aa9135e2bdb17ec4e506a7ffc35384f60 Oct 30 16:47:23.295264 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:23.295225 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:47:23.295352 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:23.295318 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:47:23.295414 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:23.295356 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:47:23.875307 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:23.875264 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" event={"ID":"69254306-affa-4f98-a1a0-93c0e29c14ae","Type":"ContainerStarted","Data":"95c397d46b637a08e031934f0b569949f844b2983e3de282b6aa6eb51db87c3e"} Oct 30 16:47:23.875470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:23.875328 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" event={"ID":"69254306-affa-4f98-a1a0-93c0e29c14ae","Type":"ContainerStarted","Data":"f806052ad1f31206d4ba151c11573d4aa9135e2bdb17ec4e506a7ffc35384f60"} Oct 30 16:47:23.936363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:23.936316 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" podStartSLOduration=1.936302094 podStartE2EDuration="1.936302094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:23.933850793 +0000 UTC m=+1474.276524834" watchObservedRunningTime="2026-10-30 16:47:23.936302094 +0000 UTC m=+1474.278976131" Oct 30 16:47:24.080625 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.075828 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" podUID="34013d8e-ff30-4d9a-87a4-13b03612b720" containerName="payload-processing" containerID="cri-o://d962f6b4e40330f6405d1989cdd9672603a90bfb828590247cf44cc115010f1c" gracePeriod=25 Oct 30 16:47:24.119864 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.118924 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" podUID="f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f" containerName="payload-pre-processing" containerID="cri-o://784ceb754bb0f2e820fa1e3a543b9cc2d9ffb2366e0b1bf24966a8aa3481cdc8" gracePeriod=25 Oct 30 16:47:24.406491 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.406421 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" Oct 30 16:47:24.430858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.430836 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" Oct 30 16:47:24.508565 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.508533 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34013d8e-ff30-4d9a-87a4-13b03612b720-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34013d8e-ff30-4d9a-87a4-13b03612b720-plugins-config-volume\") pod \"34013d8e-ff30-4d9a-87a4-13b03612b720\" (UID: \"34013d8e-ff30-4d9a-87a4-13b03612b720\") " Oct 30 16:47:24.508800 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.508778 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34013d8e-ff30-4d9a-87a4-13b03612b720-kube-api-access-xmmmx\" (UniqueName: \"kubernetes.io/projected/34013d8e-ff30-4d9a-87a4-13b03612b720-kube-api-access-xmmmx\") pod \"34013d8e-ff30-4d9a-87a4-13b03612b720\" (UID: \"34013d8e-ff30-4d9a-87a4-13b03612b720\") " Oct 30 16:47:24.508904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.508845 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f-kube-api-access-c6nmn\" (UniqueName: \"kubernetes.io/projected/f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f-kube-api-access-c6nmn\") pod \"f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f\" (UID: \"f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f\") " Oct 30 16:47:24.508904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.508893 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f-plugins-config-volume\") pod \"f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f\" (UID: \"f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f\") " Oct 30 16:47:24.508904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.508889 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34013d8e-ff30-4d9a-87a4-13b03612b720-plugins-config-volume" pod "34013d8e-ff30-4d9a-87a4-13b03612b720" (UID: "34013d8e-ff30-4d9a-87a4-13b03612b720"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:24.509350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.509325 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f-plugins-config-volume" pod "f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f" (UID: "f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:24.509545 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.509502 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34013d8e-ff30-4d9a-87a4-13b03612b720-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:24.509545 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.509527 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:24.511130 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.511110 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34013d8e-ff30-4d9a-87a4-13b03612b720-kube-api-access-xmmmx" pod "34013d8e-ff30-4d9a-87a4-13b03612b720" (UID: "34013d8e-ff30-4d9a-87a4-13b03612b720"). InnerVolumeSpecName "kube-api-access-xmmmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:24.511536 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.511518 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f-kube-api-access-c6nmn" pod "f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f" (UID: "f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f"). InnerVolumeSpecName "kube-api-access-c6nmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:24.525110 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.525084 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:24.611236 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.611203 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmmmx\" (UniqueName: \"kubernetes.io/projected/34013d8e-ff30-4d9a-87a4-13b03612b720-kube-api-access-xmmmx\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:24.611236 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.611232 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6nmn\" (UniqueName: \"kubernetes.io/projected/f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f-kube-api-access-c6nmn\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:24.864573 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.864543 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:24.869850 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.869828 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:24.882855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.882825 4861 generic.go:356] "Generic (PLEG): container finished" podID="34013d8e-ff30-4d9a-87a4-13b03612b720" containerID="d962f6b4e40330f6405d1989cdd9672603a90bfb828590247cf44cc115010f1c" exitCode=0 Oct 30 16:47:24.882973 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.882874 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" Oct 30 16:47:24.882973 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.882912 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" event={"ID":"34013d8e-ff30-4d9a-87a4-13b03612b720","Type":"ContainerDied","Data":"d962f6b4e40330f6405d1989cdd9672603a90bfb828590247cf44cc115010f1c"} Oct 30 16:47:24.882973 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.882964 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf" event={"ID":"34013d8e-ff30-4d9a-87a4-13b03612b720","Type":"ContainerDied","Data":"ab2e5f10b9acca0bf82caef7fd647e9534e39147ca15393251e3583c8673f2bb"} Oct 30 16:47:24.883100 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.882991 4861 scope.go:122] "RemoveContainer" containerID="d962f6b4e40330f6405d1989cdd9672603a90bfb828590247cf44cc115010f1c" Oct 30 16:47:24.884574 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.884542 4861 generic.go:356] "Generic (PLEG): container finished" podID="f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f" containerID="784ceb754bb0f2e820fa1e3a543b9cc2d9ffb2366e0b1bf24966a8aa3481cdc8" exitCode=0 Oct 30 16:47:24.884684 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.884588 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" Oct 30 16:47:24.884684 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.884633 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" event={"ID":"f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f","Type":"ContainerDied","Data":"784ceb754bb0f2e820fa1e3a543b9cc2d9ffb2366e0b1bf24966a8aa3481cdc8"} Oct 30 16:47:24.884684 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.884671 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v" event={"ID":"f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f","Type":"ContainerDied","Data":"53ea37becf1bd913b202fc97bbb2b86993e8d396edc57b56a4cd5c53947ad823"} Oct 30 16:47:24.885011 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.884975 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:24.885936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.885913 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:47:24.894428 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.894413 4861 scope.go:122] "RemoveContainer" containerID="d962f6b4e40330f6405d1989cdd9672603a90bfb828590247cf44cc115010f1c" Oct 30 16:47:24.894764 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:24.894736 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d962f6b4e40330f6405d1989cdd9672603a90bfb828590247cf44cc115010f1c\": container with ID starting with d962f6b4e40330f6405d1989cdd9672603a90bfb828590247cf44cc115010f1c not found: ID does not exist" containerID="d962f6b4e40330f6405d1989cdd9672603a90bfb828590247cf44cc115010f1c" Oct 30 16:47:24.894828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.894773 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d962f6b4e40330f6405d1989cdd9672603a90bfb828590247cf44cc115010f1c"} err="failed to get container status \"d962f6b4e40330f6405d1989cdd9672603a90bfb828590247cf44cc115010f1c\": rpc error: code = NotFound desc = could not find container \"d962f6b4e40330f6405d1989cdd9672603a90bfb828590247cf44cc115010f1c\": container with ID starting with d962f6b4e40330f6405d1989cdd9672603a90bfb828590247cf44cc115010f1c not found: ID does not exist" Oct 30 16:47:24.894828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.894791 4861 scope.go:122] "RemoveContainer" containerID="784ceb754bb0f2e820fa1e3a543b9cc2d9ffb2366e0b1bf24966a8aa3481cdc8" Oct 30 16:47:24.906913 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.906892 4861 scope.go:122] "RemoveContainer" containerID="784ceb754bb0f2e820fa1e3a543b9cc2d9ffb2366e0b1bf24966a8aa3481cdc8" Oct 30 16:47:24.907181 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:24.907164 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"784ceb754bb0f2e820fa1e3a543b9cc2d9ffb2366e0b1bf24966a8aa3481cdc8\": container with ID starting with 784ceb754bb0f2e820fa1e3a543b9cc2d9ffb2366e0b1bf24966a8aa3481cdc8 not found: ID does not exist" containerID="784ceb754bb0f2e820fa1e3a543b9cc2d9ffb2366e0b1bf24966a8aa3481cdc8" Oct 30 16:47:24.907258 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:24.907188 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784ceb754bb0f2e820fa1e3a543b9cc2d9ffb2366e0b1bf24966a8aa3481cdc8"} err="failed to get container status \"784ceb754bb0f2e820fa1e3a543b9cc2d9ffb2366e0b1bf24966a8aa3481cdc8\": rpc error: code = NotFound desc = could not find container \"784ceb754bb0f2e820fa1e3a543b9cc2d9ffb2366e0b1bf24966a8aa3481cdc8\": container with ID starting with 784ceb754bb0f2e820fa1e3a543b9cc2d9ffb2366e0b1bf24966a8aa3481cdc8 not found: ID does not exist" Oct 30 16:47:25.012227 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.012194 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf"] Oct 30 16:47:25.015409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.015383 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f9970a-6bfb8c4dc6-hpksf"] Oct 30 16:47:25.040177 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.040153 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v"] Oct 30 16:47:25.044052 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.044032 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f9970a-56dd657f4c-2d57v"] Oct 30 16:47:25.646776 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.646715 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:25.799043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.799016 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:25.809735 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.809693 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34013d8e-ff30-4d9a-87a4-13b03612b720" path="/var/lib/kubelet/pods/34013d8e-ff30-4d9a-87a4-13b03612b720/volumes" Oct 30 16:47:25.810256 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.810241 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f" path="/var/lib/kubelet/pods/f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f/volumes" Oct 30 16:47:25.906193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.897032 4861 generic.go:356] "Generic (PLEG): container finished" podID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerID="1b8f9b0624d4be8c381d1cd208566d752a17f871245576d8c365aadfab04fe65" exitCode=0 Oct 30 16:47:25.906193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.897270 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" event={"ID":"b7bfc8bc-b483-458e-8580-b32c699fd17b","Type":"ContainerDied","Data":"1b8f9b0624d4be8c381d1cd208566d752a17f871245576d8c365aadfab04fe65"} Oct 30 16:47:25.906193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.897330 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" event={"ID":"b7bfc8bc-b483-458e-8580-b32c699fd17b","Type":"ContainerDied","Data":"590e1eda550c94bb30f13eb2c8ebfb21f2219e7756e6452f195d81148bf01a3a"} Oct 30 16:47:25.906193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.897357 4861 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="590e1eda550c94bb30f13eb2c8ebfb21f2219e7756e6452f195d81148bf01a3a" Oct 30 16:47:25.906193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.897376 4861 scope.go:122] "RemoveContainer" containerID="e1c0aba7496559f921f20473391d3f74e8c5ed5bab855fc9be864c74043cb965" Oct 30 16:47:25.911375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.909870 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v"] Oct 30 16:47:25.911375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.910126 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" podUID="e5026254-8aef-43a6-b079-072241e3b7b4" containerName="istio-proxy" containerID="cri-o://08d46bbeff961651f9f4464d9419447c030091d00707bc2bbfd755ad9d381562" gracePeriod=30 Oct 30 16:47:25.923134 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.923110 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:47:25.972838 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:25.972805 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:26.068595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.068473 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-credential-socket\") pod \"b7bfc8bc-b483-458e-8580-b32c699fd17b\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " Oct 30 16:47:26.068958 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.068936 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-data\") pod \"b7bfc8bc-b483-458e-8580-b32c699fd17b\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " Oct 30 16:47:26.069177 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.069155 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7bfc8bc-b483-458e-8580-b32c699fd17b-kube-api-access-4njp4\" (UniqueName: \"kubernetes.io/projected/b7bfc8bc-b483-458e-8580-b32c699fd17b-kube-api-access-4njp4\") pod \"b7bfc8bc-b483-458e-8580-b32c699fd17b\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " Oct 30 16:47:26.069356 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.069328 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-workload-socket\") pod \"b7bfc8bc-b483-458e-8580-b32c699fd17b\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " Oct 30 16:47:26.069614 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.069582 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7bfc8bc-b483-458e-8580-b32c699fd17b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bfc8bc-b483-458e-8580-b32c699fd17b-istiod-ca-cert\") pod \"b7bfc8bc-b483-458e-8580-b32c699fd17b\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " Oct 30 16:47:26.071097 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.071074 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-podinfo\") pod \"b7bfc8bc-b483-458e-8580-b32c699fd17b\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " Oct 30 16:47:26.071310 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.071294 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-workload-certs\") pod \"b7bfc8bc-b483-458e-8580-b32c699fd17b\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " Oct 30 16:47:26.071490 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.071462 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-envoy\") pod \"b7bfc8bc-b483-458e-8580-b32c699fd17b\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " Oct 30 16:47:26.071622 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.071608 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-token\" (UniqueName: \"kubernetes.io/projected/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-token\") pod \"b7bfc8bc-b483-458e-8580-b32c699fd17b\" (UID: \"b7bfc8bc-b483-458e-8580-b32c699fd17b\") " Oct 30 16:47:26.074201 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.074175 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-credential-socket" pod "b7bfc8bc-b483-458e-8580-b32c699fd17b" (UID: "b7bfc8bc-b483-458e-8580-b32c699fd17b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.074747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.074705 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-data" pod "b7bfc8bc-b483-458e-8580-b32c699fd17b" (UID: "b7bfc8bc-b483-458e-8580-b32c699fd17b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.077983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.077956 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-workload-socket" pod "b7bfc8bc-b483-458e-8580-b32c699fd17b" (UID: "b7bfc8bc-b483-458e-8580-b32c699fd17b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.078272 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.078248 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7bfc8bc-b483-458e-8580-b32c699fd17b-istiod-ca-cert" pod "b7bfc8bc-b483-458e-8580-b32c699fd17b" (UID: "b7bfc8bc-b483-458e-8580-b32c699fd17b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:26.079992 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.079856 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-workload-certs" pod "b7bfc8bc-b483-458e-8580-b32c699fd17b" (UID: "b7bfc8bc-b483-458e-8580-b32c699fd17b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.095808 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.095751 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-podinfo" pod "b7bfc8bc-b483-458e-8580-b32c699fd17b" (UID: "b7bfc8bc-b483-458e-8580-b32c699fd17b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:47:26.136127 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.135809 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-envoy" pod "b7bfc8bc-b483-458e-8580-b32c699fd17b" (UID: "b7bfc8bc-b483-458e-8580-b32c699fd17b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:26.137303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.137258 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bfc8bc-b483-458e-8580-b32c699fd17b-kube-api-access-4njp4" pod "b7bfc8bc-b483-458e-8580-b32c699fd17b" (UID: "b7bfc8bc-b483-458e-8580-b32c699fd17b"). InnerVolumeSpecName "kube-api-access-4njp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:26.144000 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.143901 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-token" pod "b7bfc8bc-b483-458e-8580-b32c699fd17b" (UID: "b7bfc8bc-b483-458e-8580-b32c699fd17b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:26.173516 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.173478 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4njp4\" (UniqueName: \"kubernetes.io/projected/b7bfc8bc-b483-458e-8580-b32c699fd17b-kube-api-access-4njp4\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.173516 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.173516 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.173828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.173530 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7bfc8bc-b483-458e-8580-b32c699fd17b-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.173828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.173543 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.173828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.173555 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.173828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.173567 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.173828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.173578 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.173828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.173592 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.173828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.173603 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7bfc8bc-b483-458e-8580-b32c699fd17b-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:26.377619 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.377589 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:26.528517 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.528492 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:26.805797 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.805695 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:26.897248 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.897209 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp"] Oct 30 16:47:26.897786 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.897712 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" podUID="7e0283d9-a9fb-40a1-873a-d66b923ee500" containerName="payload-processing" containerID="cri-o://4de093a12900c35d31282f964a2d86decb9ff34c43f3310edb8461cb9eba722b" gracePeriod=30 Oct 30 16:47:26.911978 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.911950 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc"] Oct 30 16:47:26.912330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.912145 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" podUID="5159a2ef-ff89-4345-bcd5-b76bb3b6aec9" containerName="payload-pre-processing" containerID="cri-o://7f279cebe680bf812d515a0214681381604ed1748a345d47d10dbdb1c6329e25" gracePeriod=30 Oct 30 16:47:26.924099 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:26.924057 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h" Oct 30 16:47:27.029632 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.029583 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:27.090922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.090777 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h"] Oct 30 16:47:27.095548 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.095513 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4a4c68-openshift-default-5868986f-j6h8h"] Oct 30 16:47:27.179570 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.179493 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:27.461215 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.456525 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" Oct 30 16:47:27.475248 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.475222 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" Oct 30 16:47:27.498361 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.498278 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:27.504848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.504821 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0283d9-a9fb-40a1-873a-d66b923ee500-kube-api-access-t5xlh\" (UniqueName: \"kubernetes.io/projected/7e0283d9-a9fb-40a1-873a-d66b923ee500-kube-api-access-t5xlh\") pod \"7e0283d9-a9fb-40a1-873a-d66b923ee500\" (UID: \"7e0283d9-a9fb-40a1-873a-d66b923ee500\") " Oct 30 16:47:27.505079 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.504871 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5159a2ef-ff89-4345-bcd5-b76bb3b6aec9-kube-api-access-5mkjl\" (UniqueName: \"kubernetes.io/projected/5159a2ef-ff89-4345-bcd5-b76bb3b6aec9-kube-api-access-5mkjl\") pod \"5159a2ef-ff89-4345-bcd5-b76bb3b6aec9\" (UID: \"5159a2ef-ff89-4345-bcd5-b76bb3b6aec9\") " Oct 30 16:47:27.505147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.505084 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0283d9-a9fb-40a1-873a-d66b923ee500-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e0283d9-a9fb-40a1-873a-d66b923ee500-plugins-config-volume\") pod \"7e0283d9-a9fb-40a1-873a-d66b923ee500\" (UID: \"7e0283d9-a9fb-40a1-873a-d66b923ee500\") " Oct 30 16:47:27.505201 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.505191 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5159a2ef-ff89-4345-bcd5-b76bb3b6aec9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5159a2ef-ff89-4345-bcd5-b76bb3b6aec9-plugins-config-volume\") pod \"5159a2ef-ff89-4345-bcd5-b76bb3b6aec9\" (UID: \"5159a2ef-ff89-4345-bcd5-b76bb3b6aec9\") " Oct 30 16:47:27.505854 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.505826 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5159a2ef-ff89-4345-bcd5-b76bb3b6aec9-plugins-config-volume" pod "5159a2ef-ff89-4345-bcd5-b76bb3b6aec9" (UID: "5159a2ef-ff89-4345-bcd5-b76bb3b6aec9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:27.506097 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.506078 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0283d9-a9fb-40a1-873a-d66b923ee500-plugins-config-volume" pod "7e0283d9-a9fb-40a1-873a-d66b923ee500" (UID: "7e0283d9-a9fb-40a1-873a-d66b923ee500"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:27.507639 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.507616 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5159a2ef-ff89-4345-bcd5-b76bb3b6aec9-kube-api-access-5mkjl" pod "5159a2ef-ff89-4345-bcd5-b76bb3b6aec9" (UID: "5159a2ef-ff89-4345-bcd5-b76bb3b6aec9"). InnerVolumeSpecName "kube-api-access-5mkjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:27.508106 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.508074 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0283d9-a9fb-40a1-873a-d66b923ee500-kube-api-access-t5xlh" pod "7e0283d9-a9fb-40a1-873a-d66b923ee500" (UID: "7e0283d9-a9fb-40a1-873a-d66b923ee500"). InnerVolumeSpecName "kube-api-access-t5xlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:27.606584 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.606548 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5xlh\" (UniqueName: \"kubernetes.io/projected/7e0283d9-a9fb-40a1-873a-d66b923ee500-kube-api-access-t5xlh\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:27.606584 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.606579 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mkjl\" (UniqueName: \"kubernetes.io/projected/5159a2ef-ff89-4345-bcd5-b76bb3b6aec9-kube-api-access-5mkjl\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:27.606584 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.606591 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e0283d9-a9fb-40a1-873a-d66b923ee500-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:27.606864 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.606604 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5159a2ef-ff89-4345-bcd5-b76bb3b6aec9-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:27.644111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.644043 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:27.698455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.698369 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" Oct 30 16:47:27.699655 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.699604 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" Oct 30 16:47:27.788537 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.788503 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:27.813157 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.813073 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7bfc8bc-b483-458e-8580-b32c699fd17b" path="/var/lib/kubelet/pods/b7bfc8bc-b483-458e-8580-b32c699fd17b/volumes" Oct 30 16:47:27.965476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.964666 4861 generic.go:356] "Generic (PLEG): container finished" podID="5159a2ef-ff89-4345-bcd5-b76bb3b6aec9" containerID="7f279cebe680bf812d515a0214681381604ed1748a345d47d10dbdb1c6329e25" exitCode=0 Oct 30 16:47:27.965476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.965025 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" event={"ID":"5159a2ef-ff89-4345-bcd5-b76bb3b6aec9","Type":"ContainerDied","Data":"7f279cebe680bf812d515a0214681381604ed1748a345d47d10dbdb1c6329e25"} Oct 30 16:47:27.965476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.965086 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" event={"ID":"5159a2ef-ff89-4345-bcd5-b76bb3b6aec9","Type":"ContainerDied","Data":"9e5f922a3df4c5c7a06ad85936f16aa7de38c3282c89765bbeaf8fade4ffd4ef"} Oct 30 16:47:27.965476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.965117 4861 scope.go:122] "RemoveContainer" containerID="7f279cebe680bf812d515a0214681381604ed1748a345d47d10dbdb1c6329e25" Oct 30 16:47:27.965476 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.965254 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc" Oct 30 16:47:27.976018 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.975986 4861 generic.go:356] "Generic (PLEG): container finished" podID="7e0283d9-a9fb-40a1-873a-d66b923ee500" containerID="4de093a12900c35d31282f964a2d86decb9ff34c43f3310edb8461cb9eba722b" exitCode=0 Oct 30 16:47:27.976821 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.976692 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" Oct 30 16:47:27.978452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.978424 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" event={"ID":"7e0283d9-a9fb-40a1-873a-d66b923ee500","Type":"ContainerDied","Data":"4de093a12900c35d31282f964a2d86decb9ff34c43f3310edb8461cb9eba722b"} Oct 30 16:47:27.978818 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:27.978695 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp" event={"ID":"7e0283d9-a9fb-40a1-873a-d66b923ee500","Type":"ContainerDied","Data":"ec5ea82083ef3ea1a1de6094cac1647c06f0b444f0c0e20e6280c90cd92cee45"} Oct 30 16:47:28.009028 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.008998 4861 scope.go:122] "RemoveContainer" containerID="7f279cebe680bf812d515a0214681381604ed1748a345d47d10dbdb1c6329e25" Oct 30 16:47:28.013116 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:28.013062 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f279cebe680bf812d515a0214681381604ed1748a345d47d10dbdb1c6329e25\": container with ID starting with 7f279cebe680bf812d515a0214681381604ed1748a345d47d10dbdb1c6329e25 not found: ID does not exist" containerID="7f279cebe680bf812d515a0214681381604ed1748a345d47d10dbdb1c6329e25" Oct 30 16:47:28.013302 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.013121 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f279cebe680bf812d515a0214681381604ed1748a345d47d10dbdb1c6329e25"} err="failed to get container status \"7f279cebe680bf812d515a0214681381604ed1748a345d47d10dbdb1c6329e25\": rpc error: code = NotFound desc = could not find container \"7f279cebe680bf812d515a0214681381604ed1748a345d47d10dbdb1c6329e25\": container with ID starting with 7f279cebe680bf812d515a0214681381604ed1748a345d47d10dbdb1c6329e25 not found: ID does not exist" Oct 30 16:47:28.013302 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.013158 4861 scope.go:122] "RemoveContainer" containerID="4de093a12900c35d31282f964a2d86decb9ff34c43f3310edb8461cb9eba722b" Oct 30 16:47:28.026042 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.026021 4861 scope.go:122] "RemoveContainer" containerID="4de093a12900c35d31282f964a2d86decb9ff34c43f3310edb8461cb9eba722b" Oct 30 16:47:28.026323 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:28.026306 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4de093a12900c35d31282f964a2d86decb9ff34c43f3310edb8461cb9eba722b\": container with ID starting with 4de093a12900c35d31282f964a2d86decb9ff34c43f3310edb8461cb9eba722b not found: ID does not exist" containerID="4de093a12900c35d31282f964a2d86decb9ff34c43f3310edb8461cb9eba722b" Oct 30 16:47:28.026404 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.026330 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4de093a12900c35d31282f964a2d86decb9ff34c43f3310edb8461cb9eba722b"} err="failed to get container status \"4de093a12900c35d31282f964a2d86decb9ff34c43f3310edb8461cb9eba722b\": rpc error: code = NotFound desc = could not find container \"4de093a12900c35d31282f964a2d86decb9ff34c43f3310edb8461cb9eba722b\": container with ID starting with 4de093a12900c35d31282f964a2d86decb9ff34c43f3310edb8461cb9eba722b not found: ID does not exist" Oct 30 16:47:28.041027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.041001 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc"] Oct 30 16:47:28.044806 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.044780 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dc1ea0-68c7cc747f-fpxpc"] Oct 30 16:47:28.078387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.078357 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp"] Oct 30 16:47:28.082669 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.082643 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dc1ea0-575bfc4d68-lkkzp"] Oct 30 16:47:28.189289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.189196 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:28.446185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.446107 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:28.580107 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.580079 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54"] Oct 30 16:47:28.580752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.580714 4861 cpu_manager.go:414] "containerMap: removing container" podUID="5159a2ef-ff89-4345-bcd5-b76bb3b6aec9" containerName="payload-pre-processing" Oct 30 16:47:28.580752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.580745 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5159a2ef-ff89-4345-bcd5-b76bb3b6aec9" containerName="payload-pre-processing" Oct 30 16:47:28.580930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.580760 4861 cpu_manager.go:414] "containerMap: removing container" podUID="34013d8e-ff30-4d9a-87a4-13b03612b720" containerName="payload-processing" Oct 30 16:47:28.580930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.580766 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34013d8e-ff30-4d9a-87a4-13b03612b720" containerName="payload-processing" Oct 30 16:47:28.580930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.580775 4861 cpu_manager.go:414] "containerMap: removing container" podUID="f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f" containerName="payload-pre-processing" Oct 30 16:47:28.580930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.580780 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f" containerName="payload-pre-processing" Oct 30 16:47:28.580930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.580799 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerName="istio-proxy" Oct 30 16:47:28.580930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.580804 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerName="istio-proxy" Oct 30 16:47:28.580930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.580818 4861 cpu_manager.go:414] "containerMap: removing container" podUID="7e0283d9-a9fb-40a1-873a-d66b923ee500" containerName="payload-processing" Oct 30 16:47:28.580930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.580824 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0283d9-a9fb-40a1-873a-d66b923ee500" containerName="payload-processing" Oct 30 16:47:28.581201 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.580956 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerName="istio-proxy" Oct 30 16:47:28.581201 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.580966 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="5159a2ef-ff89-4345-bcd5-b76bb3b6aec9" containerName="payload-pre-processing" Oct 30 16:47:28.581201 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.580976 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="34013d8e-ff30-4d9a-87a4-13b03612b720" containerName="payload-processing" Oct 30 16:47:28.581201 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.580987 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="f09deeb3-f361-44de-b4e2-aa7d2a3f2d8f" containerName="payload-pre-processing" Oct 30 16:47:28.581201 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.581004 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerName="istio-proxy" Oct 30 16:47:28.581201 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.581011 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0283d9-a9fb-40a1-873a-d66b923ee500" containerName="payload-processing" Oct 30 16:47:28.585320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.585301 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" Oct 30 16:47:28.607036 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.607001 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54"] Oct 30 16:47:28.607564 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.607542 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-569b62-dockercfg-t86qg\"" Oct 30 16:47:28.607703 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.607616 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-569b62\"" Oct 30 16:47:28.608922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.608895 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:28.623026 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.622991 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rczrd\" (UniqueName: \"kubernetes.io/projected/824cd0d9-5af5-4c68-a4c8-62ff61852acc-kube-api-access-rczrd\") pod \"payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54\" (UID: \"824cd0d9-5af5-4c68-a4c8-62ff61852acc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" Oct 30 16:47:28.623194 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.623169 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824cd0d9-5af5-4c68-a4c8-62ff61852acc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54\" (UID: \"824cd0d9-5af5-4c68-a4c8-62ff61852acc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" Oct 30 16:47:28.742797 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.738369 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824cd0d9-5af5-4c68-a4c8-62ff61852acc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54\" (UID: \"824cd0d9-5af5-4c68-a4c8-62ff61852acc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" Oct 30 16:47:28.742797 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.738588 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczrd\" (UniqueName: \"kubernetes.io/projected/824cd0d9-5af5-4c68-a4c8-62ff61852acc-kube-api-access-rczrd\") pod \"payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54\" (UID: \"824cd0d9-5af5-4c68-a4c8-62ff61852acc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" Oct 30 16:47:28.742797 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.739873 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824cd0d9-5af5-4c68-a4c8-62ff61852acc-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54\" (UID: \"824cd0d9-5af5-4c68-a4c8-62ff61852acc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" Oct 30 16:47:28.787748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.777093 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rczrd\" (UniqueName: \"kubernetes.io/projected/824cd0d9-5af5-4c68-a4c8-62ff61852acc-kube-api-access-rczrd\") pod \"payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54\" (UID: \"824cd0d9-5af5-4c68-a4c8-62ff61852acc\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" Oct 30 16:47:28.787748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.777160 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf"] Oct 30 16:47:28.787748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.779306 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerName="istio-proxy" Oct 30 16:47:28.787748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.779327 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7bfc8bc-b483-458e-8580-b32c699fd17b" containerName="istio-proxy" Oct 30 16:47:28.787748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.784128 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" Oct 30 16:47:28.823333 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.806453 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf"] Oct 30 16:47:28.899753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.899692 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" Oct 30 16:47:28.944679 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.944650 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35fb6ac-30a4-489b-827e-90831f6827d8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf\" (UID: \"c35fb6ac-30a4-489b-827e-90831f6827d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" Oct 30 16:47:28.945049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:28.945005 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xlsw\" (UniqueName: \"kubernetes.io/projected/c35fb6ac-30a4-489b-827e-90831f6827d8-kube-api-access-6xlsw\") pod \"payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf\" (UID: \"c35fb6ac-30a4-489b-827e-90831f6827d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" Oct 30 16:47:29.046572 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:29.046479 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xlsw\" (UniqueName: \"kubernetes.io/projected/c35fb6ac-30a4-489b-827e-90831f6827d8-kube-api-access-6xlsw\") pod \"payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf\" (UID: \"c35fb6ac-30a4-489b-827e-90831f6827d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" Oct 30 16:47:29.046572 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:29.046557 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35fb6ac-30a4-489b-827e-90831f6827d8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf\" (UID: \"c35fb6ac-30a4-489b-827e-90831f6827d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" Oct 30 16:47:29.047306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:29.047286 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35fb6ac-30a4-489b-827e-90831f6827d8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf\" (UID: \"c35fb6ac-30a4-489b-827e-90831f6827d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" Oct 30 16:47:29.071042 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:29.071001 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xlsw\" (UniqueName: \"kubernetes.io/projected/c35fb6ac-30a4-489b-827e-90831f6827d8-kube-api-access-6xlsw\") pod \"payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf\" (UID: \"c35fb6ac-30a4-489b-827e-90831f6827d8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" Oct 30 16:47:29.170756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:29.170714 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" Oct 30 16:47:29.211303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:29.211275 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54"] Oct 30 16:47:29.214018 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:47:29.213968 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824cd0d9_5af5_4c68_a4c8_62ff61852acc.slice/crio-17978c4aa8af90bf6d5d838433ff2842de4294b0f68d04d1d5447f8c0f510510 WatchSource:0}: Error finding container 17978c4aa8af90bf6d5d838433ff2842de4294b0f68d04d1d5447f8c0f510510: Status 404 returned error can't find the container with id 17978c4aa8af90bf6d5d838433ff2842de4294b0f68d04d1d5447f8c0f510510 Oct 30 16:47:29.497004 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:29.496972 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf"] Oct 30 16:47:29.498253 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:47:29.498222 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35fb6ac_30a4_489b_827e_90831f6827d8.slice/crio-ed5b30650fe317c3a558ca787b1aea7639d70adae7926763a4f856a55046e6cf WatchSource:0}: Error finding container ed5b30650fe317c3a558ca787b1aea7639d70adae7926763a4f856a55046e6cf: Status 404 returned error can't find the container with id ed5b30650fe317c3a558ca787b1aea7639d70adae7926763a4f856a55046e6cf Oct 30 16:47:29.808316 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:29.808236 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5159a2ef-ff89-4345-bcd5-b76bb3b6aec9" path="/var/lib/kubelet/pods/5159a2ef-ff89-4345-bcd5-b76bb3b6aec9/volumes" Oct 30 16:47:29.808759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:29.808715 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0283d9-a9fb-40a1-873a-d66b923ee500" path="/var/lib/kubelet/pods/7e0283d9-a9fb-40a1-873a-d66b923ee500/volumes" Oct 30 16:47:29.924893 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:29.924864 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:30.020989 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:30.020951 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" event={"ID":"824cd0d9-5af5-4c68-a4c8-62ff61852acc","Type":"ContainerStarted","Data":"3be7e0d7828064286a3cbdcc9ae51a416a5e440487ba04cf8aaace557b54709b"} Oct 30 16:47:30.021151 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:30.021019 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" event={"ID":"824cd0d9-5af5-4c68-a4c8-62ff61852acc","Type":"ContainerStarted","Data":"17978c4aa8af90bf6d5d838433ff2842de4294b0f68d04d1d5447f8c0f510510"} Oct 30 16:47:30.021151 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:30.021089 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" Oct 30 16:47:30.023662 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:30.023632 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" event={"ID":"c35fb6ac-30a4-489b-827e-90831f6827d8","Type":"ContainerStarted","Data":"4c1e43473b8348da52eb260f72e3da580b724b460143702d7fa04250968de64d"} Oct 30 16:47:30.023837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:30.023676 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" event={"ID":"c35fb6ac-30a4-489b-827e-90831f6827d8","Type":"ContainerStarted","Data":"ed5b30650fe317c3a558ca787b1aea7639d70adae7926763a4f856a55046e6cf"} Oct 30 16:47:30.023837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:30.023782 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" Oct 30 16:47:30.065426 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:30.065331 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" podStartSLOduration=2.065318208 podStartE2EDuration="2.065318208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:30.062631918 +0000 UTC m=+1480.405305957" watchObservedRunningTime="2026-10-30 16:47:30.065318208 +0000 UTC m=+1480.407992300" Oct 30 16:47:30.106627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:30.106570 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" podStartSLOduration=2.106557302 podStartE2EDuration="2.106557302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:30.10389005 +0000 UTC m=+1480.446564089" watchObservedRunningTime="2026-10-30 16:47:30.106557302 +0000 UTC m=+1480.449231339" Oct 30 16:47:31.032673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.032641 4861 generic.go:356] "Generic (PLEG): container finished" podID="e5026254-8aef-43a6-b079-072241e3b7b4" containerID="08d46bbeff961651f9f4464d9419447c030091d00707bc2bbfd755ad9d381562" exitCode=0 Oct 30 16:47:31.033092 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.032713 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" event={"ID":"e5026254-8aef-43a6-b079-072241e3b7b4","Type":"ContainerDied","Data":"08d46bbeff961651f9f4464d9419447c030091d00707bc2bbfd755ad9d381562"} Oct 30 16:47:31.033092 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.032802 4861 scope.go:122] "RemoveContainer" containerID="0b2315a01b98298ca15952da55947fe27d96b433e97250ac1632c8889c65e916" Oct 30 16:47:31.600929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.600901 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:47:31.692314 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.692240 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5026254-8aef-43a6-b079-072241e3b7b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5026254-8aef-43a6-b079-072241e3b7b4-istiod-ca-cert\") pod \"e5026254-8aef-43a6-b079-072241e3b7b4\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " Oct 30 16:47:31.692314 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.692303 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-workload-socket\") pod \"e5026254-8aef-43a6-b079-072241e3b7b4\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " Oct 30 16:47:31.692502 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.692330 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-credential-socket\") pod \"e5026254-8aef-43a6-b079-072241e3b7b4\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " Oct 30 16:47:31.692502 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.692381 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5026254-8aef-43a6-b079-072241e3b7b4-kube-api-access-j2v4l\" (UniqueName: \"kubernetes.io/projected/e5026254-8aef-43a6-b079-072241e3b7b4-kube-api-access-j2v4l\") pod \"e5026254-8aef-43a6-b079-072241e3b7b4\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " Oct 30 16:47:31.692502 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.692415 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-istio-data\") pod \"e5026254-8aef-43a6-b079-072241e3b7b4\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " Oct 30 16:47:31.692666 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.692584 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-workload-socket" pod "e5026254-8aef-43a6-b079-072241e3b7b4" (UID: "e5026254-8aef-43a6-b079-072241e3b7b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:31.692666 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.692596 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-workload-certs\") pod \"e5026254-8aef-43a6-b079-072241e3b7b4\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " Oct 30 16:47:31.692666 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.692651 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-istio-envoy\") pod \"e5026254-8aef-43a6-b079-072241e3b7b4\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " Oct 30 16:47:31.692859 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.692711 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5026254-8aef-43a6-b079-072241e3b7b4-istio-token\" (UniqueName: \"kubernetes.io/projected/e5026254-8aef-43a6-b079-072241e3b7b4-istio-token\") pod \"e5026254-8aef-43a6-b079-072241e3b7b4\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " Oct 30 16:47:31.692859 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.692758 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-credential-socket" pod "e5026254-8aef-43a6-b079-072241e3b7b4" (UID: "e5026254-8aef-43a6-b079-072241e3b7b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:31.692859 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.692803 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5026254-8aef-43a6-b079-072241e3b7b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5026254-8aef-43a6-b079-072241e3b7b4-istio-podinfo\") pod \"e5026254-8aef-43a6-b079-072241e3b7b4\" (UID: \"e5026254-8aef-43a6-b079-072241e3b7b4\") " Oct 30 16:47:31.693034 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.692812 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5026254-8aef-43a6-b079-072241e3b7b4-istiod-ca-cert" pod "e5026254-8aef-43a6-b079-072241e3b7b4" (UID: "e5026254-8aef-43a6-b079-072241e3b7b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:31.693034 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.692923 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-workload-certs" pod "e5026254-8aef-43a6-b079-072241e3b7b4" (UID: "e5026254-8aef-43a6-b079-072241e3b7b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:31.693140 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.693037 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-istio-data" pod "e5026254-8aef-43a6-b079-072241e3b7b4" (UID: "e5026254-8aef-43a6-b079-072241e3b7b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:31.693635 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.693608 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:31.693635 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.693637 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5026254-8aef-43a6-b079-072241e3b7b4-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:31.693890 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.693653 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:31.693890 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.693668 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:31.693890 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.693682 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:31.695297 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.695269 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5026254-8aef-43a6-b079-072241e3b7b4-kube-api-access-j2v4l" pod "e5026254-8aef-43a6-b079-072241e3b7b4" (UID: "e5026254-8aef-43a6-b079-072241e3b7b4"). InnerVolumeSpecName "kube-api-access-j2v4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:31.695610 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.695581 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5026254-8aef-43a6-b079-072241e3b7b4-istio-podinfo" pod "e5026254-8aef-43a6-b079-072241e3b7b4" (UID: "e5026254-8aef-43a6-b079-072241e3b7b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:47:31.695748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.695621 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5026254-8aef-43a6-b079-072241e3b7b4-istio-token" pod "e5026254-8aef-43a6-b079-072241e3b7b4" (UID: "e5026254-8aef-43a6-b079-072241e3b7b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:31.695748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.695644 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-istio-envoy" pod "e5026254-8aef-43a6-b079-072241e3b7b4" (UID: "e5026254-8aef-43a6-b079-072241e3b7b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:31.795074 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.795047 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5026254-8aef-43a6-b079-072241e3b7b4-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:31.795074 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.795067 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5026254-8aef-43a6-b079-072241e3b7b4-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:31.795271 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.795081 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5026254-8aef-43a6-b079-072241e3b7b4-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:31.795271 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:31.795093 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2v4l\" (UniqueName: \"kubernetes.io/projected/e5026254-8aef-43a6-b079-072241e3b7b4-kube-api-access-j2v4l\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:32.041529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:32.041497 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" Oct 30 16:47:32.042016 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:32.041526 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v" event={"ID":"e5026254-8aef-43a6-b079-072241e3b7b4","Type":"ContainerDied","Data":"1c41ab7df25c9154e65c6630ca30db6f60281e1e3771400296b588bcf0fdf22a"} Oct 30 16:47:32.042016 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:32.041584 4861 scope.go:122] "RemoveContainer" containerID="08d46bbeff961651f9f4464d9419447c030091d00707bc2bbfd755ad9d381562" Oct 30 16:47:32.096271 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:32.096246 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v"] Oct 30 16:47:32.101796 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:32.101777 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c23a4-openshift-default-795b975d75-jgs9v"] Oct 30 16:47:32.437626 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:32.437549 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:33.807737 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:33.807696 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5026254-8aef-43a6-b079-072241e3b7b4" path="/var/lib/kubelet/pods/e5026254-8aef-43a6-b079-072241e3b7b4/volumes" Oct 30 16:47:34.368065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:34.355389 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:34.922928 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:34.922897 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:35.070933 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:35.070904 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:35.190019 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:35.189935 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w"] Oct 30 16:47:35.190202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:35.190178 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" podUID="7409c5f8-d599-4876-be1a-f5b0a25c0416" containerName="istio-proxy" containerID="cri-o://93c17f851b02924708e3049a38af1af69a4ab802dfd9f3c2e202010926b16021" gracePeriod=30 Oct 30 16:47:35.250091 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:35.250060 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:35.336749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:35.334405 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gp6rg_a2319b60-ec05-4f37-b97a-79de3c482d58/payload-processing/0.log" Oct 30 16:47:35.539529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:35.539504 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gp6rg_a2319b60-ec05-4f37-b97a-79de3c482d58/payload-processing/0.log" Oct 30 16:47:35.734261 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:35.734228 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz_1916010e-81c5-4927-af79-d4d9b343ace4/payload-processing/0.log" Oct 30 16:47:36.758059 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:36.758032 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:36.907394 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:36.907352 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:37.146590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.146514 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k"] Oct 30 16:47:37.147198 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.147180 4861 cpu_manager.go:414] "containerMap: removing container" podUID="e5026254-8aef-43a6-b079-072241e3b7b4" containerName="istio-proxy" Oct 30 16:47:37.147280 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.147198 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5026254-8aef-43a6-b079-072241e3b7b4" containerName="istio-proxy" Oct 30 16:47:37.147280 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.147215 4861 cpu_manager.go:414] "containerMap: removing container" podUID="e5026254-8aef-43a6-b079-072241e3b7b4" containerName="istio-proxy" Oct 30 16:47:37.147280 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.147220 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5026254-8aef-43a6-b079-072241e3b7b4" containerName="istio-proxy" Oct 30 16:47:37.147420 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.147333 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5026254-8aef-43a6-b079-072241e3b7b4" containerName="istio-proxy" Oct 30 16:47:37.152315 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.152294 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.163554 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.163521 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-data\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.163554 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.163549 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-token\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.163745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.163644 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-credential-socket\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.163745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.163667 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-envoy\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.163851 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.163774 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-workload-certs\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.163904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.163891 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-workload-socket\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.163939 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.163926 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-668xc\" (UniqueName: \"kubernetes.io/projected/b5f3e0b2-750b-4545-9c66-dfc005023077-kube-api-access-668xc\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.163972 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.163959 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-podinfo\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.164104 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.164092 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f3e0b2-750b-4545-9c66-dfc005023077-istiod-ca-cert\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.171519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.171501 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-76cccb-openshift-default-dockercfg-smv7q\"" Oct 30 16:47:37.175436 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.175416 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k"] Oct 30 16:47:37.216266 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.216235 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:37.285549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.281739 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f3e0b2-750b-4545-9c66-dfc005023077-istiod-ca-cert\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.285549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.281801 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-data\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.285549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.281830 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-token\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.285549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.281906 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-credential-socket\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.285549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.281932 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-envoy\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.285549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.282001 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-workload-certs\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.285549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.282124 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-workload-socket\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.285549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.282155 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-668xc\" (UniqueName: \"kubernetes.io/projected/b5f3e0b2-750b-4545-9c66-dfc005023077-kube-api-access-668xc\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.285549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.282217 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-podinfo\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.326749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.315406 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f3e0b2-750b-4545-9c66-dfc005023077-istiod-ca-cert\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.327016 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.319363 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-podinfo\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.327176 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.320524 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-workload-socket\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.332834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.320704 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-workload-certs\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.333023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.321222 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-data\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.333171 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.321597 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-credential-socket\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.333298 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.332068 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-envoy\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.365748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.358097 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-668xc\" (UniqueName: \"kubernetes.io/projected/b5f3e0b2-750b-4545-9c66-dfc005023077-kube-api-access-668xc\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.365748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.358903 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-token\") pod \"e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.377793 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.374219 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:37.470183 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.468080 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:37.709560 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.709490 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq"] Oct 30 16:47:37.710663 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.710442 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5026254-8aef-43a6-b079-072241e3b7b4" containerName="istio-proxy" Oct 30 16:47:37.716203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.716182 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.737578 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.737547 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq"] Oct 30 16:47:37.739336 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.739307 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-504442\"/\"openshift-service-ca.crt\"" Oct 30 16:47:37.742097 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.742074 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-504442\"/\"default-dockercfg-vzx5s\"" Oct 30 16:47:37.742240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.742220 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-504442\"/\"kube-root-ca.crt\"" Oct 30 16:47:37.742331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.742317 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-504442\"/\"ipp-route-504442-kserve-self-signed-certs\"" Oct 30 16:47:37.796546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.796513 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:37.796903 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.796570 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/579c56b7-c963-4ced-b809-1e40ff5fcbf5-tls-certs\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.796903 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.796661 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-model-cache\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.796903 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.796770 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xf5h\" (UniqueName: \"kubernetes.io/projected/579c56b7-c963-4ced-b809-1e40ff5fcbf5-kube-api-access-4xf5h\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.796903 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.796832 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-dshm\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.796903 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.796896 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-tmp-dir\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.797069 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.796961 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-home\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.825019 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.824997 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k"] Oct 30 16:47:37.825507 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:47:37.825484 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f3e0b2_750b_4545_9c66_dfc005023077.slice/crio-97f9d1724d8b273e1762d5749ff69dd43cf7e652d13b6edcdfba7c975f317437 WatchSource:0}: Error finding container 97f9d1724d8b273e1762d5749ff69dd43cf7e652d13b6edcdfba7c975f317437: Status 404 returned error can't find the container with id 97f9d1724d8b273e1762d5749ff69dd43cf7e652d13b6edcdfba7c975f317437 Oct 30 16:47:37.827926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.827895 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:47:37.827995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.827958 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:47:37.827995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.827984 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:47:37.898464 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.898435 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-tmp-dir\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.898563 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.898502 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-home\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.898632 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.898610 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/579c56b7-c963-4ced-b809-1e40ff5fcbf5-tls-certs\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.898696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.898678 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-model-cache\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.898978 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.898873 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xf5h\" (UniqueName: \"kubernetes.io/projected/579c56b7-c963-4ced-b809-1e40ff5fcbf5-kube-api-access-4xf5h\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.898978 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.898914 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-dshm\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.898978 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.898969 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-home\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.899191 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.899052 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-tmp-dir\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.899191 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.899080 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-model-cache\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.901426 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.901407 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/579c56b7-c963-4ced-b809-1e40ff5fcbf5-tls-certs\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.901517 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.901502 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-dshm\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:37.925665 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:37.925637 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xf5h\" (UniqueName: \"kubernetes.io/projected/579c56b7-c963-4ced-b809-1e40ff5fcbf5-kube-api-access-4xf5h\") pod \"ipp-route-504442-kserve-69995f5fc7-bkjvq\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:38.036851 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:38.035991 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:38.152571 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:38.152532 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" event={"ID":"b5f3e0b2-750b-4545-9c66-dfc005023077","Type":"ContainerStarted","Data":"c9659376aa2a0d6d014d9874d8e0be3857b0bbca6f16bfa835b338e7ea6e87ec"} Oct 30 16:47:38.152831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:38.152599 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" event={"ID":"b5f3e0b2-750b-4545-9c66-dfc005023077","Type":"ContainerStarted","Data":"97f9d1724d8b273e1762d5749ff69dd43cf7e652d13b6edcdfba7c975f317437"} Oct 30 16:47:38.203387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:38.202577 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" podStartSLOduration=1.202558484 podStartE2EDuration="1.202558484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:38.201986849 +0000 UTC m=+1488.544660882" watchObservedRunningTime="2026-10-30 16:47:38.202558484 +0000 UTC m=+1488.545232522" Oct 30 16:47:38.445243 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:38.445211 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq"] Oct 30 16:47:38.446913 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:47:38.446882 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod579c56b7_c963_4ced_b809_1e40ff5fcbf5.slice/crio-477da58ebee192e5b755c1480149d0d1840e12fa298302a3d804339c38c9da56 WatchSource:0}: Error finding container 477da58ebee192e5b755c1480149d0d1840e12fa298302a3d804339c38c9da56: Status 404 returned error can't find the container with id 477da58ebee192e5b755c1480149d0d1840e12fa298302a3d804339c38c9da56 Oct 30 16:47:38.469010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:38.468989 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:38.474795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:38.474774 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:38.889609 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:38.889572 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:39.161596 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:39.161504 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" event={"ID":"579c56b7-c963-4ced-b809-1e40ff5fcbf5","Type":"ContainerStarted","Data":"0efeb7fc47bc828bfe02027318c1d109e5d24bae68e9ad706b461b20c2276f93"} Oct 30 16:47:39.161596 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:39.161551 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" event={"ID":"579c56b7-c963-4ced-b809-1e40ff5fcbf5","Type":"ContainerStarted","Data":"477da58ebee192e5b755c1480149d0d1840e12fa298302a3d804339c38c9da56"} Oct 30 16:47:39.162058 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:39.162035 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:39.162969 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:39.162949 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:47:39.205717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:39.205677 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" podStartSLOduration=2.205665941 podStartE2EDuration="2.205665941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:39.204361523 +0000 UTC m=+1489.547035561" watchObservedRunningTime="2026-10-30 16:47:39.205665941 +0000 UTC m=+1489.548339978" Oct 30 16:47:40.037410 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.037382 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:40.495458 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.495426 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:47:40.579427 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.579337 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-workload-certs\") pod \"7409c5f8-d599-4876-be1a-f5b0a25c0416\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " Oct 30 16:47:40.579427 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.579385 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7409c5f8-d599-4876-be1a-f5b0a25c0416-kube-api-access-nbwdt\" (UniqueName: \"kubernetes.io/projected/7409c5f8-d599-4876-be1a-f5b0a25c0416-kube-api-access-nbwdt\") pod \"7409c5f8-d599-4876-be1a-f5b0a25c0416\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " Oct 30 16:47:40.579661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.579427 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-envoy\") pod \"7409c5f8-d599-4876-be1a-f5b0a25c0416\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " Oct 30 16:47:40.579661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.579465 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-credential-socket\") pod \"7409c5f8-d599-4876-be1a-f5b0a25c0416\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " Oct 30 16:47:40.579661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.579506 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-token\" (UniqueName: \"kubernetes.io/projected/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-token\") pod \"7409c5f8-d599-4876-be1a-f5b0a25c0416\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " Oct 30 16:47:40.579661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.579563 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-podinfo\") pod \"7409c5f8-d599-4876-be1a-f5b0a25c0416\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " Oct 30 16:47:40.579661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.579630 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7409c5f8-d599-4876-be1a-f5b0a25c0416-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7409c5f8-d599-4876-be1a-f5b0a25c0416-istiod-ca-cert\") pod \"7409c5f8-d599-4876-be1a-f5b0a25c0416\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " Oct 30 16:47:40.580019 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.579681 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-workload-certs" pod "7409c5f8-d599-4876-be1a-f5b0a25c0416" (UID: "7409c5f8-d599-4876-be1a-f5b0a25c0416"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:40.580019 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.579898 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-workload-socket\") pod \"7409c5f8-d599-4876-be1a-f5b0a25c0416\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " Oct 30 16:47:40.580019 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.579935 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-data\") pod \"7409c5f8-d599-4876-be1a-f5b0a25c0416\" (UID: \"7409c5f8-d599-4876-be1a-f5b0a25c0416\") " Oct 30 16:47:40.580512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.580485 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-workload-socket" pod "7409c5f8-d599-4876-be1a-f5b0a25c0416" (UID: "7409c5f8-d599-4876-be1a-f5b0a25c0416"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:40.580750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.580712 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-data" pod "7409c5f8-d599-4876-be1a-f5b0a25c0416" (UID: "7409c5f8-d599-4876-be1a-f5b0a25c0416"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:40.580850 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.580829 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-credential-socket" pod "7409c5f8-d599-4876-be1a-f5b0a25c0416" (UID: "7409c5f8-d599-4876-be1a-f5b0a25c0416"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:40.580940 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.580837 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7409c5f8-d599-4876-be1a-f5b0a25c0416-istiod-ca-cert" pod "7409c5f8-d599-4876-be1a-f5b0a25c0416" (UID: "7409c5f8-d599-4876-be1a-f5b0a25c0416"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:40.580997 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.580966 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.580997 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.580985 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.583069 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.583042 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-envoy" pod "7409c5f8-d599-4876-be1a-f5b0a25c0416" (UID: "7409c5f8-d599-4876-be1a-f5b0a25c0416"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:47:40.583194 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.583172 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-token" pod "7409c5f8-d599-4876-be1a-f5b0a25c0416" (UID: "7409c5f8-d599-4876-be1a-f5b0a25c0416"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:40.583409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.583386 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7409c5f8-d599-4876-be1a-f5b0a25c0416-kube-api-access-nbwdt" pod "7409c5f8-d599-4876-be1a-f5b0a25c0416" (UID: "7409c5f8-d599-4876-be1a-f5b0a25c0416"). InnerVolumeSpecName "kube-api-access-nbwdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:40.583669 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.583651 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-podinfo" pod "7409c5f8-d599-4876-be1a-f5b0a25c0416" (UID: "7409c5f8-d599-4876-be1a-f5b0a25c0416"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:47:40.682405 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.682376 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.682405 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.682402 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.682573 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.682412 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.682573 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.682420 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.682573 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.682428 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7409c5f8-d599-4876-be1a-f5b0a25c0416-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.682573 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.682436 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7409c5f8-d599-4876-be1a-f5b0a25c0416-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.682573 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.682444 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbwdt\" (UniqueName: \"kubernetes.io/projected/7409c5f8-d599-4876-be1a-f5b0a25c0416-kube-api-access-nbwdt\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:40.740110 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.740075 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:40.887796 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:40.887155 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:41.044938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:41.043775 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" Oct 30 16:47:41.044938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:41.043945 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" Oct 30 16:47:41.179843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:41.179714 4861 generic.go:356] "Generic (PLEG): container finished" podID="7409c5f8-d599-4876-be1a-f5b0a25c0416" containerID="93c17f851b02924708e3049a38af1af69a4ab802dfd9f3c2e202010926b16021" exitCode=0 Oct 30 16:47:41.179993 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:41.179908 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" event={"ID":"7409c5f8-d599-4876-be1a-f5b0a25c0416","Type":"ContainerDied","Data":"93c17f851b02924708e3049a38af1af69a4ab802dfd9f3c2e202010926b16021"} Oct 30 16:47:41.179993 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:41.179962 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" Oct 30 16:47:41.180111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:41.180015 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w" event={"ID":"7409c5f8-d599-4876-be1a-f5b0a25c0416","Type":"ContainerDied","Data":"1d8937d87a83182a483f0659cbd3fc7006227f9d6b0bf3e39df013c413620f9d"} Oct 30 16:47:41.180111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:41.180059 4861 scope.go:122] "RemoveContainer" containerID="93c17f851b02924708e3049a38af1af69a4ab802dfd9f3c2e202010926b16021" Oct 30 16:47:41.192978 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:41.192963 4861 scope.go:122] "RemoveContainer" containerID="b18f190561fb3f4d2e9d115f847faa32d71712b3c3781dca6fa74f38b155542e" Oct 30 16:47:41.207536 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:41.207515 4861 scope.go:122] "RemoveContainer" containerID="93c17f851b02924708e3049a38af1af69a4ab802dfd9f3c2e202010926b16021" Oct 30 16:47:41.207853 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:41.207829 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c17f851b02924708e3049a38af1af69a4ab802dfd9f3c2e202010926b16021\": container with ID starting with 93c17f851b02924708e3049a38af1af69a4ab802dfd9f3c2e202010926b16021 not found: ID does not exist" containerID="93c17f851b02924708e3049a38af1af69a4ab802dfd9f3c2e202010926b16021" Oct 30 16:47:41.207928 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:41.207860 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c17f851b02924708e3049a38af1af69a4ab802dfd9f3c2e202010926b16021"} err="failed to get container status \"93c17f851b02924708e3049a38af1af69a4ab802dfd9f3c2e202010926b16021\": rpc error: code = NotFound desc = could not find container \"93c17f851b02924708e3049a38af1af69a4ab802dfd9f3c2e202010926b16021\": container with ID starting with 93c17f851b02924708e3049a38af1af69a4ab802dfd9f3c2e202010926b16021 not found: ID does not exist" Oct 30 16:47:41.207928 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:41.207877 4861 scope.go:122] "RemoveContainer" containerID="b18f190561fb3f4d2e9d115f847faa32d71712b3c3781dca6fa74f38b155542e" Oct 30 16:47:41.208179 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:41.208156 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18f190561fb3f4d2e9d115f847faa32d71712b3c3781dca6fa74f38b155542e\": container with ID starting with b18f190561fb3f4d2e9d115f847faa32d71712b3c3781dca6fa74f38b155542e not found: ID does not exist" containerID="b18f190561fb3f4d2e9d115f847faa32d71712b3c3781dca6fa74f38b155542e" Oct 30 16:47:41.208229 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:41.208191 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18f190561fb3f4d2e9d115f847faa32d71712b3c3781dca6fa74f38b155542e"} err="failed to get container status \"b18f190561fb3f4d2e9d115f847faa32d71712b3c3781dca6fa74f38b155542e\": rpc error: code = NotFound desc = could not find container \"b18f190561fb3f4d2e9d115f847faa32d71712b3c3781dca6fa74f38b155542e\": container with ID starting with b18f190561fb3f4d2e9d115f847faa32d71712b3c3781dca6fa74f38b155542e not found: ID does not exist" Oct 30 16:47:41.246142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:41.246099 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w"] Oct 30 16:47:41.249788 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:41.249763 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-dc1ea0-openshift-default-77f566f56f-8tc6w"] Oct 30 16:47:41.808835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:41.808804 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7409c5f8-d599-4876-be1a-f5b0a25c0416" path="/var/lib/kubelet/pods/7409c5f8-d599-4876-be1a-f5b0a25c0416/volumes" Oct 30 16:47:42.434855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.434822 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:42.571418 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.571383 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz"] Oct 30 16:47:42.572015 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.572001 4861 cpu_manager.go:414] "containerMap: removing container" podUID="7409c5f8-d599-4876-be1a-f5b0a25c0416" containerName="istio-proxy" Oct 30 16:47:42.572015 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.572016 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7409c5f8-d599-4876-be1a-f5b0a25c0416" containerName="istio-proxy" Oct 30 16:47:42.572117 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.572051 4861 cpu_manager.go:414] "containerMap: removing container" podUID="7409c5f8-d599-4876-be1a-f5b0a25c0416" containerName="istio-proxy" Oct 30 16:47:42.572117 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.572057 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7409c5f8-d599-4876-be1a-f5b0a25c0416" containerName="istio-proxy" Oct 30 16:47:42.572179 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.572142 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="7409c5f8-d599-4876-be1a-f5b0a25c0416" containerName="istio-proxy" Oct 30 16:47:42.577169 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.577148 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" Oct 30 16:47:42.607624 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.600115 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz"] Oct 30 16:47:42.622549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.622521 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-76cccb\"" Oct 30 16:47:42.623206 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.623176 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-76cccb-dockercfg-x85dm\"" Oct 30 16:47:42.623484 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.623460 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:42.735690 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.735642 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a952e9-7858-45b7-8d4d-8cccfae8ca3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz\" (UID: \"b9a952e9-7858-45b7-8d4d-8cccfae8ca3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" Oct 30 16:47:42.736059 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.736033 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8q9z\" (UniqueName: \"kubernetes.io/projected/b9a952e9-7858-45b7-8d4d-8cccfae8ca3c-kube-api-access-m8q9z\") pod \"payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz\" (UID: \"b9a952e9-7858-45b7-8d4d-8cccfae8ca3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" Oct 30 16:47:42.756841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.756791 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z"] Oct 30 16:47:42.758209 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.758165 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="7409c5f8-d599-4876-be1a-f5b0a25c0416" containerName="istio-proxy" Oct 30 16:47:42.796306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.795047 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z"] Oct 30 16:47:42.796306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.795175 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" Oct 30 16:47:42.858763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.851028 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5prmc\" (UniqueName: \"kubernetes.io/projected/a18fbfaf-05e6-40a5-8853-7465bd9e92b0-kube-api-access-5prmc\") pod \"payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z\" (UID: \"a18fbfaf-05e6-40a5-8853-7465bd9e92b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" Oct 30 16:47:42.858763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.851243 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a952e9-7858-45b7-8d4d-8cccfae8ca3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz\" (UID: \"b9a952e9-7858-45b7-8d4d-8cccfae8ca3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" Oct 30 16:47:42.858763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.851347 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8q9z\" (UniqueName: \"kubernetes.io/projected/b9a952e9-7858-45b7-8d4d-8cccfae8ca3c-kube-api-access-m8q9z\") pod \"payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz\" (UID: \"b9a952e9-7858-45b7-8d4d-8cccfae8ca3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" Oct 30 16:47:42.858763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.851457 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18fbfaf-05e6-40a5-8853-7465bd9e92b0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z\" (UID: \"a18fbfaf-05e6-40a5-8853-7465bd9e92b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" Oct 30 16:47:42.858763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.852612 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a952e9-7858-45b7-8d4d-8cccfae8ca3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz\" (UID: \"b9a952e9-7858-45b7-8d4d-8cccfae8ca3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" Oct 30 16:47:42.889635 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.889600 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8q9z\" (UniqueName: \"kubernetes.io/projected/b9a952e9-7858-45b7-8d4d-8cccfae8ca3c-kube-api-access-m8q9z\") pod \"payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz\" (UID: \"b9a952e9-7858-45b7-8d4d-8cccfae8ca3c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" Oct 30 16:47:42.891865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.891843 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" Oct 30 16:47:42.892271 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.892246 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:42.966837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.961614 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18fbfaf-05e6-40a5-8853-7465bd9e92b0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z\" (UID: \"a18fbfaf-05e6-40a5-8853-7465bd9e92b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" Oct 30 16:47:42.966837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.962434 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18fbfaf-05e6-40a5-8853-7465bd9e92b0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z\" (UID: \"a18fbfaf-05e6-40a5-8853-7465bd9e92b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" Oct 30 16:47:42.966837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:42.962559 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5prmc\" (UniqueName: \"kubernetes.io/projected/a18fbfaf-05e6-40a5-8853-7465bd9e92b0-kube-api-access-5prmc\") pod \"payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z\" (UID: \"a18fbfaf-05e6-40a5-8853-7465bd9e92b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" Oct 30 16:47:43.005790 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:43.005698 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5prmc\" (UniqueName: \"kubernetes.io/projected/a18fbfaf-05e6-40a5-8853-7465bd9e92b0-kube-api-access-5prmc\") pod \"payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z\" (UID: \"a18fbfaf-05e6-40a5-8853-7465bd9e92b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" Oct 30 16:47:43.205645 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:43.205600 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" Oct 30 16:47:43.273609 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:43.273579 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz"] Oct 30 16:47:43.519973 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:43.519948 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z"] Oct 30 16:47:43.521270 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:47:43.521243 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18fbfaf_05e6_40a5_8853_7465bd9e92b0.slice/crio-f17c563dd2d2b5a464671130c35b52c25740f8f8dfc9ca9ae4f0f7dd52531de8 WatchSource:0}: Error finding container f17c563dd2d2b5a464671130c35b52c25740f8f8dfc9ca9ae4f0f7dd52531de8: Status 404 returned error can't find the container with id f17c563dd2d2b5a464671130c35b52c25740f8f8dfc9ca9ae4f0f7dd52531de8 Oct 30 16:47:43.680157 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:43.680127 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:44.203543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:44.203513 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" event={"ID":"b9a952e9-7858-45b7-8d4d-8cccfae8ca3c","Type":"ContainerStarted","Data":"370435ffe293fac8b9eb6846e55912cb319396edd43a52d79801d7f9a34b8e6a"} Oct 30 16:47:44.203702 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:44.203565 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" event={"ID":"b9a952e9-7858-45b7-8d4d-8cccfae8ca3c","Type":"ContainerStarted","Data":"d7b1f9a9fb04a9f8879b681ce17a0b5e02eb73cf09df1fc7b9c1653753f120da"} Oct 30 16:47:44.203702 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:44.203625 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" Oct 30 16:47:44.205332 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:44.205308 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" event={"ID":"a18fbfaf-05e6-40a5-8853-7465bd9e92b0","Type":"ContainerStarted","Data":"e9bae21fb4bdba828b899137b97997cca063f3dd89ab814e1200c0163e1f4986"} Oct 30 16:47:44.205434 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:44.205352 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" event={"ID":"a18fbfaf-05e6-40a5-8853-7465bd9e92b0","Type":"ContainerStarted","Data":"f17c563dd2d2b5a464671130c35b52c25740f8f8dfc9ca9ae4f0f7dd52531de8"} Oct 30 16:47:44.205434 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:44.205378 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" Oct 30 16:47:44.246430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:44.246365 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" podStartSLOduration=2.246348182 podStartE2EDuration="2.246348182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:44.244952598 +0000 UTC m=+1494.587626637" watchObservedRunningTime="2026-10-30 16:47:44.246348182 +0000 UTC m=+1494.589022224" Oct 30 16:47:44.288613 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:44.288572 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" podStartSLOduration=2.288561542 podStartE2EDuration="2.288561542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:44.286091815 +0000 UTC m=+1494.628765854" watchObservedRunningTime="2026-10-30 16:47:44.288561542 +0000 UTC m=+1494.631235579" Oct 30 16:47:44.916287 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:44.916260 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:47.181038 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.181009 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l"] Oct 30 16:47:47.185338 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.185318 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:47:47.204083 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.204063 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-569b62-serving-cert\"" Oct 30 16:47:47.220241 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.220212 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l"] Oct 30 16:47:47.316784 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.316753 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/801f1750-fa15-4727-b81a-d7d8ff88a0d8-metrics-certs\") pod \"maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l\" (UID: \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:47:47.316948 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.316898 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7sm6\" (UniqueName: \"kubernetes.io/projected/801f1750-fa15-4727-b81a-d7d8ff88a0d8-kube-api-access-d7sm6\") pod \"maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l\" (UID: \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:47:47.317035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.316974 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/801f1750-fa15-4727-b81a-d7d8ff88a0d8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l\" (UID: \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:47:47.317114 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.317099 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/801f1750-fa15-4727-b81a-d7d8ff88a0d8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l\" (UID: \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:47:47.326456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.326431 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:47.434131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.420589 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/801f1750-fa15-4727-b81a-d7d8ff88a0d8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l\" (UID: \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:47:47.434131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.420808 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/801f1750-fa15-4727-b81a-d7d8ff88a0d8-metrics-certs\") pod \"maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l\" (UID: \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:47:47.434131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.420919 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7sm6\" (UniqueName: \"kubernetes.io/projected/801f1750-fa15-4727-b81a-d7d8ff88a0d8-kube-api-access-d7sm6\") pod \"maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l\" (UID: \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:47:47.434131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.420966 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/801f1750-fa15-4727-b81a-d7d8ff88a0d8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l\" (UID: \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:47:47.434131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.421786 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/801f1750-fa15-4727-b81a-d7d8ff88a0d8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l\" (UID: \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:47:47.471583 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.450020 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/801f1750-fa15-4727-b81a-d7d8ff88a0d8-metrics-certs\") pod \"maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l\" (UID: \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:47:47.471583 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.463426 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7sm6\" (UniqueName: \"kubernetes.io/projected/801f1750-fa15-4727-b81a-d7d8ff88a0d8-kube-api-access-d7sm6\") pod \"maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l\" (UID: \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:47:47.472504 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.472474 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/801f1750-fa15-4727-b81a-d7d8ff88a0d8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l\" (UID: \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:47:47.478755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.473441 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:47.508395 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.506997 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:47:47.850401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:47.850378 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l"] Oct 30 16:47:47.851261 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:47:47.851233 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801f1750_fa15_4727_b81a_d7d8ff88a0d8.slice/crio-ffc497352241021e4d205203d08f27cce9911536436de52639ede4f7df49e856 WatchSource:0}: Error finding container ffc497352241021e4d205203d08f27cce9911536436de52639ede4f7df49e856: Status 404 returned error can't find the container with id ffc497352241021e4d205203d08f27cce9911536436de52639ede4f7df49e856 Oct 30 16:47:48.037588 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:48.037553 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:48.050817 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:48.050785 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:48.237014 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:48.236978 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" event={"ID":"801f1750-fa15-4727-b81a-d7d8ff88a0d8","Type":"ContainerStarted","Data":"ffc497352241021e4d205203d08f27cce9911536436de52639ede4f7df49e856"} Oct 30 16:47:48.249047 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:48.249021 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:47:49.247969 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:49.247936 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" event={"ID":"801f1750-fa15-4727-b81a-d7d8ff88a0d8","Type":"ContainerStarted","Data":"153f2944ca31b26350948ead5b68768b1b6722fe5e06ab8569f2534e53fc616a"} Oct 30 16:47:49.248451 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:49.248148 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:47:49.270686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:49.270661 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:49.319819 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:49.309972 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" podStartSLOduration=1.893292675 podStartE2EDuration="2.309951573s" totalImagesPullingTime="416.658898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:49.306958408 +0000 UTC m=+1499.649632446" watchObservedRunningTime="2026-10-30 16:47:49.309951573 +0000 UTC m=+1499.652625612" Oct 30 16:47:50.034657 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:50.034632 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:47:50.037009 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:50.036980 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:47:50.290560 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:50.290477 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:47:50.940834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:50.940796 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj"] Oct 30 16:47:50.941112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:50.941086 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" podUID="acd472a2-210b-425b-80c3-0fa8ee391ca8" containerName="maas-api" containerID="cri-o://4d53cfecaf9361f7758641fe953f8a067c3b0f615e50e78150961925c7925233" gracePeriod=30 Oct 30 16:47:51.036331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.036302 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:51.228923 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.228894 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:51.244172 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.244150 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:47:51.265941 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.265914 4861 generic.go:356] "Generic (PLEG): container finished" podID="acd472a2-210b-425b-80c3-0fa8ee391ca8" containerID="4d53cfecaf9361f7758641fe953f8a067c3b0f615e50e78150961925c7925233" exitCode=0 Oct 30 16:47:51.266047 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.265955 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" Oct 30 16:47:51.266047 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.265998 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" event={"ID":"acd472a2-210b-425b-80c3-0fa8ee391ca8","Type":"ContainerDied","Data":"4d53cfecaf9361f7758641fe953f8a067c3b0f615e50e78150961925c7925233"} Oct 30 16:47:51.266140 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.266055 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj" event={"ID":"acd472a2-210b-425b-80c3-0fa8ee391ca8","Type":"ContainerDied","Data":"f4cd385cc5d245bb2be1ccd7f310fd1af5f07ecf643003cd68767bbde16bb1bd"} Oct 30 16:47:51.266140 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.266090 4861 scope.go:122] "RemoveContainer" containerID="4d53cfecaf9361f7758641fe953f8a067c3b0f615e50e78150961925c7925233" Oct 30 16:47:51.276191 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.276175 4861 scope.go:122] "RemoveContainer" containerID="4d53cfecaf9361f7758641fe953f8a067c3b0f615e50e78150961925c7925233" Oct 30 16:47:51.276538 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:47:51.276517 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d53cfecaf9361f7758641fe953f8a067c3b0f615e50e78150961925c7925233\": container with ID starting with 4d53cfecaf9361f7758641fe953f8a067c3b0f615e50e78150961925c7925233 not found: ID does not exist" containerID="4d53cfecaf9361f7758641fe953f8a067c3b0f615e50e78150961925c7925233" Oct 30 16:47:51.276621 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.276546 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d53cfecaf9361f7758641fe953f8a067c3b0f615e50e78150961925c7925233"} err="failed to get container status \"4d53cfecaf9361f7758641fe953f8a067c3b0f615e50e78150961925c7925233\": rpc error: code = NotFound desc = could not find container \"4d53cfecaf9361f7758641fe953f8a067c3b0f615e50e78150961925c7925233\": container with ID starting with 4d53cfecaf9361f7758641fe953f8a067c3b0f615e50e78150961925c7925233 not found: ID does not exist" Oct 30 16:47:51.374530 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.374502 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:51.387935 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.387906 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd472a2-210b-425b-80c3-0fa8ee391ca8-kube-api-access-nx6q4\" (UniqueName: \"kubernetes.io/projected/acd472a2-210b-425b-80c3-0fa8ee391ca8-kube-api-access-nx6q4\") pod \"acd472a2-210b-425b-80c3-0fa8ee391ca8\" (UID: \"acd472a2-210b-425b-80c3-0fa8ee391ca8\") " Oct 30 16:47:51.388084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.388062 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acd472a2-210b-425b-80c3-0fa8ee391ca8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/acd472a2-210b-425b-80c3-0fa8ee391ca8-metrics-certs\") pod \"acd472a2-210b-425b-80c3-0fa8ee391ca8\" (UID: \"acd472a2-210b-425b-80c3-0fa8ee391ca8\") " Oct 30 16:47:51.388160 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.388138 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acd472a2-210b-425b-80c3-0fa8ee391ca8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acd472a2-210b-425b-80c3-0fa8ee391ca8-maas-api-tls\") pod \"acd472a2-210b-425b-80c3-0fa8ee391ca8\" (UID: \"acd472a2-210b-425b-80c3-0fa8ee391ca8\") " Oct 30 16:47:51.388248 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.388174 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd472a2-210b-425b-80c3-0fa8ee391ca8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd472a2-210b-425b-80c3-0fa8ee391ca8-openshift-service-ca\") pod \"acd472a2-210b-425b-80c3-0fa8ee391ca8\" (UID: \"acd472a2-210b-425b-80c3-0fa8ee391ca8\") " Oct 30 16:47:51.388881 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.388759 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd472a2-210b-425b-80c3-0fa8ee391ca8-openshift-service-ca" pod "acd472a2-210b-425b-80c3-0fa8ee391ca8" (UID: "acd472a2-210b-425b-80c3-0fa8ee391ca8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:47:51.389099 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.389009 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd472a2-210b-425b-80c3-0fa8ee391ca8-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:51.390450 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.390431 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd472a2-210b-425b-80c3-0fa8ee391ca8-maas-api-tls" pod "acd472a2-210b-425b-80c3-0fa8ee391ca8" (UID: "acd472a2-210b-425b-80c3-0fa8ee391ca8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:51.390546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.390532 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd472a2-210b-425b-80c3-0fa8ee391ca8-kube-api-access-nx6q4" pod "acd472a2-210b-425b-80c3-0fa8ee391ca8" (UID: "acd472a2-210b-425b-80c3-0fa8ee391ca8"). InnerVolumeSpecName "kube-api-access-nx6q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:47:51.391010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.390988 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acd472a2-210b-425b-80c3-0fa8ee391ca8-metrics-certs" pod "acd472a2-210b-425b-80c3-0fa8ee391ca8" (UID: "acd472a2-210b-425b-80c3-0fa8ee391ca8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:47:51.491748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.491130 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx6q4\" (UniqueName: \"kubernetes.io/projected/acd472a2-210b-425b-80c3-0fa8ee391ca8-kube-api-access-nx6q4\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:51.491748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.491168 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acd472a2-210b-425b-80c3-0fa8ee391ca8-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:51.491748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.491183 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acd472a2-210b-425b-80c3-0fa8ee391ca8-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:47:51.654671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.654639 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj"] Oct 30 16:47:51.667601 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.667574 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-64c409-9c5487c6c-b8zfj"] Oct 30 16:47:51.726504 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.726447 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:51.811127 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.811039 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd472a2-210b-425b-80c3-0fa8ee391ca8" path="/var/lib/kubelet/pods/acd472a2-210b-425b-80c3-0fa8ee391ca8/volumes" Oct 30 16:47:51.866851 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:51.866824 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:52.005956 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:52.005919 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:52.370669 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:52.370639 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:53.889602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:53.888831 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:55.217396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:55.217367 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" Oct 30 16:47:55.217914 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:55.217696 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" Oct 30 16:47:55.261832 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:55.261807 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:47:56.338686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:56.338654 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:58.166753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:58.162436 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:58.780995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:58.780966 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:58.941973 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:58.941941 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:59.089236 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:59.089143 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27"] Oct 30 16:47:59.089444 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:59.089416 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" podUID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" containerName="istio-proxy" containerID="cri-o://6847f4e934c31493a57f35bdb55ed2b2f3381427976215e2cf46cfbed9317c25" gracePeriod=30 Oct 30 16:47:59.165386 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:59.165357 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:47:59.738782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:47:59.738745 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:00.356397 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.356359 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:00.397175 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.397145 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k"] Oct 30 16:48:00.397788 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.397773 4861 cpu_manager.go:414] "containerMap: removing container" podUID="acd472a2-210b-425b-80c3-0fa8ee391ca8" containerName="maas-api" Oct 30 16:48:00.397846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.397788 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd472a2-210b-425b-80c3-0fa8ee391ca8" containerName="maas-api" Oct 30 16:48:00.397917 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.397908 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd472a2-210b-425b-80c3-0fa8ee391ca8" containerName="maas-api" Oct 30 16:48:00.402990 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.402972 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:48:00.428129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.428082 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-76cccb-serving-cert\"" Oct 30 16:48:00.432564 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.432536 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k"] Oct 30 16:48:00.505811 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.505784 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:00.532943 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.532913 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-maas-api-tls\") pod \"maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k\" (UID: \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:48:00.533147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.532971 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvhdj\" (UniqueName: \"kubernetes.io/projected/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-kube-api-access-dvhdj\") pod \"maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k\" (UID: \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:48:00.533147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.533073 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k\" (UID: \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:48:00.533147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.533145 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-metrics-certs\") pod \"maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k\" (UID: \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:48:00.643795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.638624 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-maas-api-tls\") pod \"maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k\" (UID: \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:48:00.643795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.638711 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvhdj\" (UniqueName: \"kubernetes.io/projected/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-kube-api-access-dvhdj\") pod \"maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k\" (UID: \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:48:00.643795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.638846 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k\" (UID: \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:48:00.643795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.638923 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-metrics-certs\") pod \"maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k\" (UID: \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:48:00.645083 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.645044 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k\" (UID: \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:48:00.656751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.656654 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-maas-api-tls\") pod \"maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k\" (UID: \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:48:00.658529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.658495 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:00.666248 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.666216 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvhdj\" (UniqueName: \"kubernetes.io/projected/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-kube-api-access-dvhdj\") pod \"maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k\" (UID: \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:48:00.682589 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.682545 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-metrics-certs\") pod \"maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k\" (UID: \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:48:00.719045 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.718980 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:48:00.914754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:00.841271 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:01.019236 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.019181 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:01.055795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.055763 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k"] Oct 30 16:48:01.056394 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:48:01.056373 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5eba3b4_3bc9_4065_8714_4ae82e8cf790.slice/crio-a10981520478e2f962f53d17ff4cba5f77e00c2e8aeb8a9be0df241408c89e3f WatchSource:0}: Error finding container a10981520478e2f962f53d17ff4cba5f77e00c2e8aeb8a9be0df241408c89e3f: Status 404 returned error can't find the container with id a10981520478e2f962f53d17ff4cba5f77e00c2e8aeb8a9be0df241408c89e3f Oct 30 16:48:01.104068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.104042 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj"] Oct 30 16:48:01.116782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.110070 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.136031 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.136007 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-72b379-openshift-default-dockercfg-2d56l\"" Oct 30 16:48:01.138608 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.138585 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj"] Oct 30 16:48:01.180393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.180325 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:01.252750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.251871 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db4fb72-7b88-4237-8c83-38f14f803caa-istiod-ca-cert\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.260834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.260784 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-workload-certs\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.261129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.261109 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-envoy\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.262288 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.261353 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-podinfo\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.262572 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.262535 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-workload-socket\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.262833 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.262803 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-credential-socket\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.263030 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.262992 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-token\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.263269 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.263250 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2cn6\" (UniqueName: \"kubernetes.io/projected/9db4fb72-7b88-4237-8c83-38f14f803caa-kube-api-access-h2cn6\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.263471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.263455 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-data\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.376240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.376213 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-workload-certs\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.376936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.376910 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-envoy\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.377090 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.376968 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-podinfo\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.377090 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.376990 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-workload-socket\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.377090 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.377030 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-credential-socket\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.377090 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.377055 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-token\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.377237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.377102 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2cn6\" (UniqueName: \"kubernetes.io/projected/9db4fb72-7b88-4237-8c83-38f14f803caa-kube-api-access-h2cn6\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.377237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.377134 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-data\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.377237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.377199 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db4fb72-7b88-4237-8c83-38f14f803caa-istiod-ca-cert\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.377560 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.377543 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-workload-socket\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.377787 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.377768 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db4fb72-7b88-4237-8c83-38f14f803caa-istiod-ca-cert\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.377860 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.376917 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-workload-certs\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.378063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.378015 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-credential-socket\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.378287 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.378262 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-data\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.380767 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.380417 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-envoy\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.381311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.381283 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" event={"ID":"f5eba3b4-3bc9-4065-8714-4ae82e8cf790","Type":"ContainerStarted","Data":"a10981520478e2f962f53d17ff4cba5f77e00c2e8aeb8a9be0df241408c89e3f"} Oct 30 16:48:01.381854 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.381837 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-podinfo\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.402374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.402348 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-token\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.402587 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.402567 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2cn6\" (UniqueName: \"kubernetes.io/projected/9db4fb72-7b88-4237-8c83-38f14f803caa-kube-api-access-h2cn6\") pod \"e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.427471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.427446 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:01.752154 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.752123 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:01.932795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.925119 4861 patch_prober.go:29] interesting pod/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:48:01.932795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:01.925183 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" podUID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:48:02.004184 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:02.003045 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj"] Oct 30 16:48:02.006827 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:48:02.006797 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db4fb72_7b88_4237_8c83_38f14f803caa.slice/crio-0e193f36c26863e369ebb221958fe77675f81fd5ee8f7f156eff91a9903c84c6 WatchSource:0}: Error finding container 0e193f36c26863e369ebb221958fe77675f81fd5ee8f7f156eff91a9903c84c6: Status 404 returned error can't find the container with id 0e193f36c26863e369ebb221958fe77675f81fd5ee8f7f156eff91a9903c84c6 Oct 30 16:48:02.012913 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:02.009980 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:48:02.012913 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:02.010061 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:48:02.012913 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:02.010102 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:48:02.399631 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:02.399594 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" event={"ID":"9db4fb72-7b88-4237-8c83-38f14f803caa","Type":"ContainerStarted","Data":"1dfbad26e9dff50ec551681a222dccd3b0708f50681f62c23176ca48295f4457"} Oct 30 16:48:02.399976 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:02.399657 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" event={"ID":"9db4fb72-7b88-4237-8c83-38f14f803caa","Type":"ContainerStarted","Data":"0e193f36c26863e369ebb221958fe77675f81fd5ee8f7f156eff91a9903c84c6"} Oct 30 16:48:02.403807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:02.402912 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" event={"ID":"f5eba3b4-3bc9-4065-8714-4ae82e8cf790","Type":"ContainerStarted","Data":"e83cf738c9053c395805abbb4d9f9aced4266459acfe74b930458dcb7e59c464"} Oct 30 16:48:02.403807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:02.403679 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:48:02.428241 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:02.428198 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:02.447326 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:02.447270 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" podStartSLOduration=1.447252335 podStartE2EDuration="1.447252335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:02.447157146 +0000 UTC m=+1512.789831184" watchObservedRunningTime="2026-10-30 16:48:02.447252335 +0000 UTC m=+1512.789926374" Oct 30 16:48:02.497113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:02.497037 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" podStartSLOduration=2.10921831 podStartE2EDuration="2.497018851s" totalImagesPullingTime="387.800541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:02.494820974 +0000 UTC m=+1512.837495018" watchObservedRunningTime="2026-10-30 16:48:02.497018851 +0000 UTC m=+1512.839692889" Oct 30 16:48:03.116892 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:03.116845 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:03.434112 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:03.434041 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:04.381876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.381851 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:48:04.416675 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.416648 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-envoy\") pod \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " Oct 30 16:48:04.416847 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.416697 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-workload-certs\") pod \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " Oct 30 16:48:04.416847 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.416772 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-credential-socket\") pod \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " Oct 30 16:48:04.416847 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.416802 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-podinfo\") pod \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " Oct 30 16:48:04.417027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.416853 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istiod-ca-cert\") pod \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " Oct 30 16:48:04.417027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.416904 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-data\") pod \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " Oct 30 16:48:04.417027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.416950 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-workload-socket\") pod \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " Oct 30 16:48:04.417174 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.417051 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-workload-certs" pod "68fec9d9-15ed-45c2-95d3-fce8d047a6db" (UID: "68fec9d9-15ed-45c2-95d3-fce8d047a6db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:04.417174 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.417138 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68fec9d9-15ed-45c2-95d3-fce8d047a6db-kube-api-access-tkqsp\" (UniqueName: \"kubernetes.io/projected/68fec9d9-15ed-45c2-95d3-fce8d047a6db-kube-api-access-tkqsp\") pod \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " Oct 30 16:48:04.417279 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.417192 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-token\" (UniqueName: \"kubernetes.io/projected/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-token\") pod \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\" (UID: \"68fec9d9-15ed-45c2-95d3-fce8d047a6db\") " Oct 30 16:48:04.417331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.417279 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-data" pod "68fec9d9-15ed-45c2-95d3-fce8d047a6db" (UID: "68fec9d9-15ed-45c2-95d3-fce8d047a6db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:04.417331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.417294 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istiod-ca-cert" pod "68fec9d9-15ed-45c2-95d3-fce8d047a6db" (UID: "68fec9d9-15ed-45c2-95d3-fce8d047a6db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:04.417861 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.417469 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-workload-socket" pod "68fec9d9-15ed-45c2-95d3-fce8d047a6db" (UID: "68fec9d9-15ed-45c2-95d3-fce8d047a6db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:04.417861 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.417801 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-credential-socket" pod "68fec9d9-15ed-45c2-95d3-fce8d047a6db" (UID: "68fec9d9-15ed-45c2-95d3-fce8d047a6db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:04.417861 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.417859 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:04.418068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.417880 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:04.418068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.417892 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:04.418068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.417903 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:04.418879 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.418855 4861 generic.go:356] "Generic (PLEG): container finished" podID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" containerID="6847f4e934c31493a57f35bdb55ed2b2f3381427976215e2cf46cfbed9317c25" exitCode=0 Oct 30 16:48:04.419045 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.419021 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" event={"ID":"68fec9d9-15ed-45c2-95d3-fce8d047a6db","Type":"ContainerDied","Data":"6847f4e934c31493a57f35bdb55ed2b2f3381427976215e2cf46cfbed9317c25"} Oct 30 16:48:04.419133 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.419080 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" event={"ID":"68fec9d9-15ed-45c2-95d3-fce8d047a6db","Type":"ContainerDied","Data":"566d1b71b3f88d02df807a4a86a2a4dbc2c6c38b8867d9b779dac40090fac666"} Oct 30 16:48:04.419133 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.419114 4861 scope.go:122] "RemoveContainer" containerID="6847f4e934c31493a57f35bdb55ed2b2f3381427976215e2cf46cfbed9317c25" Oct 30 16:48:04.419928 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.419323 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27" Oct 30 16:48:04.419928 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.419422 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-podinfo" pod "68fec9d9-15ed-45c2-95d3-fce8d047a6db" (UID: "68fec9d9-15ed-45c2-95d3-fce8d047a6db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:48:04.419928 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.419871 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:04.419928 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.419874 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fec9d9-15ed-45c2-95d3-fce8d047a6db-kube-api-access-tkqsp" pod "68fec9d9-15ed-45c2-95d3-fce8d047a6db" (UID: "68fec9d9-15ed-45c2-95d3-fce8d047a6db"). InnerVolumeSpecName "kube-api-access-tkqsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:04.419928 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.419918 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-token" pod "68fec9d9-15ed-45c2-95d3-fce8d047a6db" (UID: "68fec9d9-15ed-45c2-95d3-fce8d047a6db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:04.420353 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.420326 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-envoy" pod "68fec9d9-15ed-45c2-95d3-fce8d047a6db" (UID: "68fec9d9-15ed-45c2-95d3-fce8d047a6db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:04.421055 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.421038 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:48:04.452091 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.452074 4861 scope.go:122] "RemoveContainer" containerID="1107a716eb0df91b1649fa75a12456e4914cba6a0e27b9c9fe729450f876b864" Oct 30 16:48:04.463316 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.463292 4861 scope.go:122] "RemoveContainer" containerID="6847f4e934c31493a57f35bdb55ed2b2f3381427976215e2cf46cfbed9317c25" Oct 30 16:48:04.463654 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:48:04.463624 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6847f4e934c31493a57f35bdb55ed2b2f3381427976215e2cf46cfbed9317c25\": container with ID starting with 6847f4e934c31493a57f35bdb55ed2b2f3381427976215e2cf46cfbed9317c25 not found: ID does not exist" containerID="6847f4e934c31493a57f35bdb55ed2b2f3381427976215e2cf46cfbed9317c25" Oct 30 16:48:04.463791 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.463664 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6847f4e934c31493a57f35bdb55ed2b2f3381427976215e2cf46cfbed9317c25"} err="failed to get container status \"6847f4e934c31493a57f35bdb55ed2b2f3381427976215e2cf46cfbed9317c25\": rpc error: code = NotFound desc = could not find container \"6847f4e934c31493a57f35bdb55ed2b2f3381427976215e2cf46cfbed9317c25\": container with ID starting with 6847f4e934c31493a57f35bdb55ed2b2f3381427976215e2cf46cfbed9317c25 not found: ID does not exist" Oct 30 16:48:04.463791 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.463689 4861 scope.go:122] "RemoveContainer" containerID="1107a716eb0df91b1649fa75a12456e4914cba6a0e27b9c9fe729450f876b864" Oct 30 16:48:04.464165 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:48:04.464141 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1107a716eb0df91b1649fa75a12456e4914cba6a0e27b9c9fe729450f876b864\": container with ID starting with 1107a716eb0df91b1649fa75a12456e4914cba6a0e27b9c9fe729450f876b864 not found: ID does not exist" containerID="1107a716eb0df91b1649fa75a12456e4914cba6a0e27b9c9fe729450f876b864" Oct 30 16:48:04.464261 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.464173 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1107a716eb0df91b1649fa75a12456e4914cba6a0e27b9c9fe729450f876b864"} err="failed to get container status \"1107a716eb0df91b1649fa75a12456e4914cba6a0e27b9c9fe729450f876b864\": rpc error: code = NotFound desc = could not find container \"1107a716eb0df91b1649fa75a12456e4914cba6a0e27b9c9fe729450f876b864\": container with ID starting with 1107a716eb0df91b1649fa75a12456e4914cba6a0e27b9c9fe729450f876b864 not found: ID does not exist" Oct 30 16:48:04.519845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.519765 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:04.520014 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.519993 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68fec9d9-15ed-45c2-95d3-fce8d047a6db-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:04.520080 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.520014 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:04.520080 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.520030 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkqsp\" (UniqueName: \"kubernetes.io/projected/68fec9d9-15ed-45c2-95d3-fce8d047a6db-kube-api-access-tkqsp\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:04.520080 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.520044 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68fec9d9-15ed-45c2-95d3-fce8d047a6db-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:04.792314 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.792283 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27"] Oct 30 16:48:04.804514 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:04.804491 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-64c409-openshift-default-54797cbd65-fcg27"] Oct 30 16:48:05.051988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:05.051912 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:05.204736 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:05.204702 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:05.350882 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:05.349567 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:05.807243 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:05.807211 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" path="/var/lib/kubelet/pods/68fec9d9-15ed-45c2-95d3-fce8d047a6db/volumes" Oct 30 16:48:06.767618 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:06.767583 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:06.883927 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:06.883899 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b"] Oct 30 16:48:06.884513 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:06.884498 4861 cpu_manager.go:414] "containerMap: removing container" podUID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" containerName="istio-proxy" Oct 30 16:48:06.884553 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:06.884514 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" containerName="istio-proxy" Oct 30 16:48:06.884649 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:06.884640 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" containerName="istio-proxy" Oct 30 16:48:06.889914 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:06.889892 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" Oct 30 16:48:06.909098 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:06.909075 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b"] Oct 30 16:48:06.917985 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:06.917963 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-72b379\"" Oct 30 16:48:06.918140 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:06.918122 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:06.953528 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:06.953491 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8ctj\" (UniqueName: \"kubernetes.io/projected/fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848-kube-api-access-s8ctj\") pod \"payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b\" (UID: \"fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" Oct 30 16:48:06.953695 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:06.953670 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b\" (UID: \"fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" Oct 30 16:48:07.055855 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.048689 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr"] Oct 30 16:48:07.089753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.056212 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b\" (UID: \"fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" Oct 30 16:48:07.089753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.077098 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8ctj\" (UniqueName: \"kubernetes.io/projected/fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848-kube-api-access-s8ctj\") pod \"payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b\" (UID: \"fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" Oct 30 16:48:07.089753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.057150 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b\" (UID: \"fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" Oct 30 16:48:07.089753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.077578 4861 cpu_manager.go:414] "containerMap: removing container" podUID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" containerName="istio-proxy" Oct 30 16:48:07.089753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.077598 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" containerName="istio-proxy" Oct 30 16:48:07.089753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.077976 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="68fec9d9-15ed-45c2-95d3-fce8d047a6db" containerName="istio-proxy" Oct 30 16:48:07.165356 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.164514 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr"] Oct 30 16:48:07.165356 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.164644 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" Oct 30 16:48:07.180766 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.168444 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8ctj\" (UniqueName: \"kubernetes.io/projected/fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848-kube-api-access-s8ctj\") pod \"payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b\" (UID: \"fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" Oct 30 16:48:07.181110 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.180980 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-72b379-dockercfg-gtvk5\"" Oct 30 16:48:07.220312 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.218304 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:07.257808 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.257776 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" Oct 30 16:48:07.361638 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.361448 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3ed68e-c3d2-4c25-a388-3c00d78128c1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr\" (UID: \"ad3ed68e-c3d2-4c25-a388-3c00d78128c1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" Oct 30 16:48:07.361638 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.361506 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsbtc\" (UniqueName: \"kubernetes.io/projected/ad3ed68e-c3d2-4c25-a388-3c00d78128c1-kube-api-access-hsbtc\") pod \"payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr\" (UID: \"ad3ed68e-c3d2-4c25-a388-3c00d78128c1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" Oct 30 16:48:07.462685 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.462645 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3ed68e-c3d2-4c25-a388-3c00d78128c1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr\" (UID: \"ad3ed68e-c3d2-4c25-a388-3c00d78128c1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" Oct 30 16:48:07.462877 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.462692 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsbtc\" (UniqueName: \"kubernetes.io/projected/ad3ed68e-c3d2-4c25-a388-3c00d78128c1-kube-api-access-hsbtc\") pod \"payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr\" (UID: \"ad3ed68e-c3d2-4c25-a388-3c00d78128c1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" Oct 30 16:48:07.463415 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.463397 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3ed68e-c3d2-4c25-a388-3c00d78128c1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr\" (UID: \"ad3ed68e-c3d2-4c25-a388-3c00d78128c1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" Oct 30 16:48:07.486755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.486696 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsbtc\" (UniqueName: \"kubernetes.io/projected/ad3ed68e-c3d2-4c25-a388-3c00d78128c1-kube-api-access-hsbtc\") pod \"payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr\" (UID: \"ad3ed68e-c3d2-4c25-a388-3c00d78128c1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" Oct 30 16:48:07.560839 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.560806 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" Oct 30 16:48:07.624292 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.624262 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b"] Oct 30 16:48:07.626796 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:48:07.626768 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9a9bee_f0b9_4951_aeaa_40b6f2c9d848.slice/crio-0a25c05490bd677d8212c9024aa9130ed25de2b4d3a76051aea484a48dc2cb76 WatchSource:0}: Error finding container 0a25c05490bd677d8212c9024aa9130ed25de2b4d3a76051aea484a48dc2cb76: Status 404 returned error can't find the container with id 0a25c05490bd677d8212c9024aa9130ed25de2b4d3a76051aea484a48dc2cb76 Oct 30 16:48:07.872368 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:07.872342 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr"] Oct 30 16:48:07.873372 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:48:07.873350 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3ed68e_c3d2_4c25_a388_3c00d78128c1.slice/crio-34f6bdade904f53f87d30727d667e40cc240265cbac7cb1179ee1e06a6af2482 WatchSource:0}: Error finding container 34f6bdade904f53f87d30727d667e40cc240265cbac7cb1179ee1e06a6af2482: Status 404 returned error can't find the container with id 34f6bdade904f53f87d30727d667e40cc240265cbac7cb1179ee1e06a6af2482 Oct 30 16:48:08.349969 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:08.349942 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:08.458669 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:08.458629 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" event={"ID":"ad3ed68e-c3d2-4c25-a388-3c00d78128c1","Type":"ContainerStarted","Data":"9f150c0ceb4f027cfea93f7c53ffc55bf3be6370e2d8fa1d1230a82eebac01f2"} Oct 30 16:48:08.458886 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:08.458695 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" Oct 30 16:48:08.458886 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:08.458759 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" event={"ID":"ad3ed68e-c3d2-4c25-a388-3c00d78128c1","Type":"ContainerStarted","Data":"34f6bdade904f53f87d30727d667e40cc240265cbac7cb1179ee1e06a6af2482"} Oct 30 16:48:08.460430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:08.460405 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" event={"ID":"fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848","Type":"ContainerStarted","Data":"eb48ea0da9d0c1050ddb896c5fbdd1f646fc81a20d7390145f6ee6fcf47d43cc"} Oct 30 16:48:08.460566 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:08.460444 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" event={"ID":"fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848","Type":"ContainerStarted","Data":"0a25c05490bd677d8212c9024aa9130ed25de2b4d3a76051aea484a48dc2cb76"} Oct 30 16:48:08.460566 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:08.460477 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" Oct 30 16:48:08.506022 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:08.505991 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:08.508920 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:08.508881 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" podStartSLOduration=2.508868852 podStartE2EDuration="2.508868852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:08.506306216 +0000 UTC m=+1518.848980251" watchObservedRunningTime="2026-10-30 16:48:08.508868852 +0000 UTC m=+1518.851542922" Oct 30 16:48:08.550148 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:08.550109 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" podStartSLOduration=2.550095899 podStartE2EDuration="2.550095899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:08.547891719 +0000 UTC m=+1518.890565756" watchObservedRunningTime="2026-10-30 16:48:08.550095899 +0000 UTC m=+1518.892769937" Oct 30 16:48:08.643072 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:08.642989 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm"] Oct 30 16:48:08.643245 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:08.643190 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" podUID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerName="istio-proxy" containerID="cri-o://7d8d7860238c2b67ddaa3cbdb398a6c903cfc749be5889c0b9b7e55caca6b26f" gracePeriod=30 Oct 30 16:48:08.704901 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:08.704877 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:09.425943 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:09.425916 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:48:09.582990 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:09.582955 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:09.723778 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:09.723741 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:10.147237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:10.147137 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:10.329752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:10.329124 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:11.691743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:11.691700 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:12.135175 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:12.135137 4861 patch_prober.go:29] interesting pod/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:48:12.135391 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:12.135201 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" podUID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:48:13.762333 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:13.762303 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:14.072513 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.072479 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:48:14.081750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.081696 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-kube-api-access-bnbwm\" (UniqueName: \"kubernetes.io/projected/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-kube-api-access-bnbwm\") pod \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " Oct 30 16:48:14.081846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.081775 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-workload-socket\") pod \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " Oct 30 16:48:14.081846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.081805 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-credential-socket\") pod \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " Oct 30 16:48:14.081931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.081866 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-token\" (UniqueName: \"kubernetes.io/projected/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-token\") pod \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " Oct 30 16:48:14.081931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.081888 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-workload-certs\") pod \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " Oct 30 16:48:14.082035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.081999 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-data\") pod \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " Oct 30 16:48:14.082095 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.082031 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istiod-ca-cert\") pod \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " Oct 30 16:48:14.082095 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.082036 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-workload-socket" pod "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" (UID: "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:14.082208 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.082118 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-credential-socket" pod "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" (UID: "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:14.082208 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.082138 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-envoy\") pod \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " Oct 30 16:48:14.082300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.082207 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-podinfo\") pod \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\" (UID: \"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e\") " Oct 30 16:48:14.082300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.082279 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-data" pod "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" (UID: "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:14.082370 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.082300 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-workload-certs" pod "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" (UID: "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:14.082484 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.082461 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istiod-ca-cert" pod "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" (UID: "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:14.082707 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.082691 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:14.082802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.082715 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:14.082802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.082748 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:14.082802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.082764 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:14.082802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.082778 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:14.084424 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.084293 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-kube-api-access-bnbwm" pod "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" (UID: "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e"). InnerVolumeSpecName "kube-api-access-bnbwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:14.084706 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.084687 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-podinfo" pod "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" (UID: "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:48:14.084936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.084919 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-token" pod "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" (UID: "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:14.084936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.084924 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-envoy" pod "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" (UID: "c0c27888-3ef3-4b1e-9a85-5d369cf6f92e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:14.184735 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.184692 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnbwm\" (UniqueName: \"kubernetes.io/projected/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-kube-api-access-bnbwm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:14.184876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.184751 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:14.184876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.184769 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:14.184876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.184782 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:48:14.284468 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.284432 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:14.435380 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.435347 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:14.528230 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.528194 4861 generic.go:356] "Generic (PLEG): container finished" podID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerID="7d8d7860238c2b67ddaa3cbdb398a6c903cfc749be5889c0b9b7e55caca6b26f" exitCode=0 Oct 30 16:48:14.528419 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.528258 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" Oct 30 16:48:14.528419 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.528288 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" event={"ID":"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e","Type":"ContainerDied","Data":"7d8d7860238c2b67ddaa3cbdb398a6c903cfc749be5889c0b9b7e55caca6b26f"} Oct 30 16:48:14.528419 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.528375 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm" event={"ID":"c0c27888-3ef3-4b1e-9a85-5d369cf6f92e","Type":"ContainerDied","Data":"deb82e42b125a150a9a0b69857737289a4d726c0b52ec99d5749d2eb1b4c0683"} Oct 30 16:48:14.528535 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.528409 4861 scope.go:122] "RemoveContainer" containerID="7d8d7860238c2b67ddaa3cbdb398a6c903cfc749be5889c0b9b7e55caca6b26f" Oct 30 16:48:14.541388 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.541368 4861 scope.go:122] "RemoveContainer" containerID="1e530304e02b6027c5a5de5fdb235e248ddc6011b5b5418b29087724ac1ad666" Oct 30 16:48:14.552872 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.552849 4861 scope.go:122] "RemoveContainer" containerID="7d8d7860238c2b67ddaa3cbdb398a6c903cfc749be5889c0b9b7e55caca6b26f" Oct 30 16:48:14.554895 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:48:14.554863 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d8d7860238c2b67ddaa3cbdb398a6c903cfc749be5889c0b9b7e55caca6b26f\": container with ID starting with 7d8d7860238c2b67ddaa3cbdb398a6c903cfc749be5889c0b9b7e55caca6b26f not found: ID does not exist" containerID="7d8d7860238c2b67ddaa3cbdb398a6c903cfc749be5889c0b9b7e55caca6b26f" Oct 30 16:48:14.555002 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.554906 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d8d7860238c2b67ddaa3cbdb398a6c903cfc749be5889c0b9b7e55caca6b26f"} err="failed to get container status \"7d8d7860238c2b67ddaa3cbdb398a6c903cfc749be5889c0b9b7e55caca6b26f\": rpc error: code = NotFound desc = could not find container \"7d8d7860238c2b67ddaa3cbdb398a6c903cfc749be5889c0b9b7e55caca6b26f\": container with ID starting with 7d8d7860238c2b67ddaa3cbdb398a6c903cfc749be5889c0b9b7e55caca6b26f not found: ID does not exist" Oct 30 16:48:14.555002 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.554932 4861 scope.go:122] "RemoveContainer" containerID="1e530304e02b6027c5a5de5fdb235e248ddc6011b5b5418b29087724ac1ad666" Oct 30 16:48:14.555497 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:48:14.555474 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e530304e02b6027c5a5de5fdb235e248ddc6011b5b5418b29087724ac1ad666\": container with ID starting with 1e530304e02b6027c5a5de5fdb235e248ddc6011b5b5418b29087724ac1ad666 not found: ID does not exist" containerID="1e530304e02b6027c5a5de5fdb235e248ddc6011b5b5418b29087724ac1ad666" Oct 30 16:48:14.555587 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.555498 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e530304e02b6027c5a5de5fdb235e248ddc6011b5b5418b29087724ac1ad666"} err="failed to get container status \"1e530304e02b6027c5a5de5fdb235e248ddc6011b5b5418b29087724ac1ad666\": rpc error: code = NotFound desc = could not find container \"1e530304e02b6027c5a5de5fdb235e248ddc6011b5b5418b29087724ac1ad666\": container with ID starting with 1e530304e02b6027c5a5de5fdb235e248ddc6011b5b5418b29087724ac1ad666 not found: ID does not exist" Oct 30 16:48:14.582535 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.582500 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm"] Oct 30 16:48:14.587129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:14.587105 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f9970a-openshift-default-5759bbc9c9-w7rhm"] Oct 30 16:48:15.831636 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:15.831595 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" path="/var/lib/kubelet/pods/c0c27888-3ef3-4b1e-9a85-5d369cf6f92e/volumes" Oct 30 16:48:16.032487 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.032456 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:16.163180 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.163100 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg"] Oct 30 16:48:16.163733 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.163705 4861 cpu_manager.go:414] "containerMap: removing container" podUID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerName="istio-proxy" Oct 30 16:48:16.163790 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.163733 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerName="istio-proxy" Oct 30 16:48:16.163888 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.163879 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerName="istio-proxy" Oct 30 16:48:16.168798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.168773 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" Oct 30 16:48:16.183203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.183181 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:16.196419 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.196386 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-cfe9f0-dockercfg-ds5w7\"" Oct 30 16:48:16.233456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.233426 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg"] Oct 30 16:48:16.234438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.234408 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng5x2\" (UniqueName: \"kubernetes.io/projected/5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2-kube-api-access-ng5x2\") pod \"payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg\" (UID: \"5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" Oct 30 16:48:16.234613 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.234461 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg\" (UID: \"5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" Oct 30 16:48:16.264303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.264266 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-cfe9f0\"" Oct 30 16:48:16.326829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.326783 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57"] Oct 30 16:48:16.370765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.349713 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng5x2\" (UniqueName: \"kubernetes.io/projected/5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2-kube-api-access-ng5x2\") pod \"payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg\" (UID: \"5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" Oct 30 16:48:16.370765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.349827 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg\" (UID: \"5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" Oct 30 16:48:16.370765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.350907 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg\" (UID: \"5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" Oct 30 16:48:16.440362 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.440281 4861 cpu_manager.go:414] "containerMap: removing container" podUID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerName="istio-proxy" Oct 30 16:48:16.440594 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.440573 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerName="istio-proxy" Oct 30 16:48:16.441005 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.440975 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c27888-3ef3-4b1e-9a85-5d369cf6f92e" containerName="istio-proxy" Oct 30 16:48:16.444851 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.444814 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng5x2\" (UniqueName: \"kubernetes.io/projected/5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2-kube-api-access-ng5x2\") pod \"payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg\" (UID: \"5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" Oct 30 16:48:16.446556 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.446538 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" Oct 30 16:48:16.448442 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.448370 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57"] Oct 30 16:48:16.498751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.488816 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" Oct 30 16:48:16.556698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.556665 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcjfc\" (UniqueName: \"kubernetes.io/projected/936e6d9b-be30-4ea6-90e1-bcc3787fe99a-kube-api-access-fcjfc\") pod \"payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57\" (UID: \"936e6d9b-be30-4ea6-90e1-bcc3787fe99a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" Oct 30 16:48:16.557254 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.557231 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936e6d9b-be30-4ea6-90e1-bcc3787fe99a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57\" (UID: \"936e6d9b-be30-4ea6-90e1-bcc3787fe99a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" Oct 30 16:48:16.658919 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.658889 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936e6d9b-be30-4ea6-90e1-bcc3787fe99a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57\" (UID: \"936e6d9b-be30-4ea6-90e1-bcc3787fe99a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" Oct 30 16:48:16.659072 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.658961 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcjfc\" (UniqueName: \"kubernetes.io/projected/936e6d9b-be30-4ea6-90e1-bcc3787fe99a-kube-api-access-fcjfc\") pod \"payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57\" (UID: \"936e6d9b-be30-4ea6-90e1-bcc3787fe99a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" Oct 30 16:48:16.659924 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.659905 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936e6d9b-be30-4ea6-90e1-bcc3787fe99a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57\" (UID: \"936e6d9b-be30-4ea6-90e1-bcc3787fe99a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" Oct 30 16:48:16.682811 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.682786 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcjfc\" (UniqueName: \"kubernetes.io/projected/936e6d9b-be30-4ea6-90e1-bcc3787fe99a-kube-api-access-fcjfc\") pod \"payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57\" (UID: \"936e6d9b-be30-4ea6-90e1-bcc3787fe99a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" Oct 30 16:48:16.863877 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.863850 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" Oct 30 16:48:16.885682 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:16.885658 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg"] Oct 30 16:48:16.886428 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:48:16.886390 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a895fbb_1e0e_4c0d_bf08_48de9c92e1f2.slice/crio-44262f1b3afa5d7e838459f8d29b5ac474133f641369f18a9d690f07f5154109 WatchSource:0}: Error finding container 44262f1b3afa5d7e838459f8d29b5ac474133f641369f18a9d690f07f5154109: Status 404 returned error can't find the container with id 44262f1b3afa5d7e838459f8d29b5ac474133f641369f18a9d690f07f5154109 Oct 30 16:48:17.240337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:17.240300 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:17.241347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:17.241324 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57"] Oct 30 16:48:17.574225 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:17.574185 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" event={"ID":"936e6d9b-be30-4ea6-90e1-bcc3787fe99a","Type":"ContainerStarted","Data":"7f3f43f199df15be0874d772ca2f29cddd4b071a8b0b8d2fbf0787aa21b33ca3"} Oct 30 16:48:17.574403 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:17.574253 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" Oct 30 16:48:17.574403 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:17.574280 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" event={"ID":"936e6d9b-be30-4ea6-90e1-bcc3787fe99a","Type":"ContainerStarted","Data":"fc73b88d8945491caa00f381ad8ada3f88b91b4c6689b87b41ee8adce0e3ae11"} Oct 30 16:48:17.575868 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:17.575841 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" event={"ID":"5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2","Type":"ContainerStarted","Data":"1c28615d336f50d00d06ba3317c84a44953d475cd2baa7b8b64afd99795920fa"} Oct 30 16:48:17.575961 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:17.575892 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" event={"ID":"5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2","Type":"ContainerStarted","Data":"44262f1b3afa5d7e838459f8d29b5ac474133f641369f18a9d690f07f5154109"} Oct 30 16:48:17.576021 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:17.575956 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" Oct 30 16:48:17.635987 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:17.635934 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" podStartSLOduration=1.635919989 podStartE2EDuration="1.635919989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:17.631874565 +0000 UTC m=+1527.974548630" watchObservedRunningTime="2026-10-30 16:48:17.635919989 +0000 UTC m=+1527.978594026" Oct 30 16:48:17.675869 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:17.675825 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" podStartSLOduration=1.675813177 podStartE2EDuration="1.675813177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:17.675342605 +0000 UTC m=+1528.018016643" watchObservedRunningTime="2026-10-30 16:48:17.675813177 +0000 UTC m=+1528.018487214" Oct 30 16:48:18.351391 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.351354 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:18.727826 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.727791 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5"] Oct 30 16:48:18.733641 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.733612 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.758198 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.758164 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5"] Oct 30 16:48:18.765521 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.765474 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-2e9431-openshift-default-dockercfg-99dn5\"" Oct 30 16:48:18.789799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.789718 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab025762-9af1-4c25-a0cc-4ab9c0221987-istiod-ca-cert\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.790055 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.789827 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh4rw\" (UniqueName: \"kubernetes.io/projected/ab025762-9af1-4c25-a0cc-4ab9c0221987-kube-api-access-rh4rw\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.790055 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.789905 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-token\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.790055 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.789961 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-credential-socket\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.790055 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.790017 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-data\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.790055 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.790046 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-podinfo\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.790443 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.790163 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-workload-socket\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.790443 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.790199 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-workload-certs\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.790443 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.790255 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-envoy\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.808043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.808008 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:18.931747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.904884 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-token\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.931747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.904959 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-credential-socket\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.931747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.905019 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-data\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.931747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.905052 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-podinfo\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.931747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.905165 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-workload-socket\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.931747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.905199 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-workload-certs\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.931747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.905252 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-envoy\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.931747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.905341 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab025762-9af1-4c25-a0cc-4ab9c0221987-istiod-ca-cert\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.931747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.905382 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh4rw\" (UniqueName: \"kubernetes.io/projected/ab025762-9af1-4c25-a0cc-4ab9c0221987-kube-api-access-rh4rw\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.931747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.906378 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-credential-socket\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.931747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.906617 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-data\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.940326 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.939197 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-workload-certs\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.940326 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.939486 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-workload-socket\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.940326 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.940269 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab025762-9af1-4c25-a0cc-4ab9c0221987-istiod-ca-cert\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.996792 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.991715 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:18.996792 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.992834 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-token\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:18.996792 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:18.993888 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-podinfo\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:19.014747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:19.007496 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-envoy\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:19.014747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:19.008086 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh4rw\" (UniqueName: \"kubernetes.io/projected/ab025762-9af1-4c25-a0cc-4ab9c0221987-kube-api-access-rh4rw\") pod \"e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:19.075754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:19.067048 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:19.439400 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:48:19.439372 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab025762_9af1_4c25_a0cc_4ab9c0221987.slice/crio-0f4ba2809fc77f0ea57e5fd21aa30f8da40affb8013a3caed2fc60fd09e64639 WatchSource:0}: Error finding container 0f4ba2809fc77f0ea57e5fd21aa30f8da40affb8013a3caed2fc60fd09e64639: Status 404 returned error can't find the container with id 0f4ba2809fc77f0ea57e5fd21aa30f8da40affb8013a3caed2fc60fd09e64639 Oct 30 16:48:19.441365 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:19.441339 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5"] Oct 30 16:48:19.442056 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:19.442029 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:48:19.442108 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:19.442085 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:48:19.442150 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:19.442114 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:48:19.471300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:19.471283 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" Oct 30 16:48:19.471398 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:19.471372 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" Oct 30 16:48:19.611413 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:19.611380 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" event={"ID":"ab025762-9af1-4c25-a0cc-4ab9c0221987","Type":"ContainerStarted","Data":"787fff535d36a91672b7d646daf82f9e95b75a9c6d6d43948e30a7472849d818"} Oct 30 16:48:19.611549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:19.611431 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" event={"ID":"ab025762-9af1-4c25-a0cc-4ab9c0221987","Type":"ContainerStarted","Data":"0f4ba2809fc77f0ea57e5fd21aa30f8da40affb8013a3caed2fc60fd09e64639"} Oct 30 16:48:19.660101 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:19.660042 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" podStartSLOduration=1.6600272569999999 podStartE2EDuration="1.660027257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:19.659140728 +0000 UTC m=+1530.001814767" watchObservedRunningTime="2026-10-30 16:48:19.660027257 +0000 UTC m=+1530.002701295" Oct 30 16:48:20.068064 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:20.068028 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:20.073986 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:20.073960 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:20.299707 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:48:20.299662 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:48:20.604049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:20.603990 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:20.660243 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:20.653767 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:20.660243 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:20.654959 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:48:21.663121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:21.663044 4861 generic.go:356] "Generic (PLEG): container finished" podID="69254306-affa-4f98-a1a0-93c0e29c14ae" containerID="95c397d46b637a08e031934f0b569949f844b2983e3de282b6aa6eb51db87c3e" exitCode=0 Oct 30 16:48:21.663121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:21.663089 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" event={"ID":"69254306-affa-4f98-a1a0-93c0e29c14ae","Type":"ContainerDied","Data":"95c397d46b637a08e031934f0b569949f844b2983e3de282b6aa6eb51db87c3e"} Oct 30 16:48:21.663659 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:21.663642 4861 scope.go:122] "RemoveContainer" containerID="95c397d46b637a08e031934f0b569949f844b2983e3de282b6aa6eb51db87c3e" Oct 30 16:48:21.665239 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:21.665208 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:48:21.665318 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:21.665300 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:48:21.665368 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:21.665353 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:48:22.163623 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:22.163530 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:22.305759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:22.305680 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:22.517777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:22.517362 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:22.673717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:22.673679 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" event={"ID":"69254306-affa-4f98-a1a0-93c0e29c14ae","Type":"ContainerStarted","Data":"627e74539b89cc3524f26942cdbcfb5f3bda6026dc169192efd99c097c0f8743"} Oct 30 16:48:22.864498 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:22.864416 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:48:22.865565 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:22.865540 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 16:48:22.865693 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:22.865590 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" podUID="69254306-affa-4f98-a1a0-93c0e29c14ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 16:48:23.887937 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:23.871832 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 16:48:23.887937 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:23.871897 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" podUID="69254306-affa-4f98-a1a0-93c0e29c14ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 16:48:24.143585 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:24.143488 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:24.326664 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:24.326629 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:24.867747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:24.864804 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 16:48:24.867747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:24.864871 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" podUID="69254306-affa-4f98-a1a0-93c0e29c14ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 16:48:24.890035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:24.889430 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:48:25.154804 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:25.153763 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:25.863966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:25.863932 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 16:48:25.864145 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:25.863979 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" podUID="69254306-affa-4f98-a1a0-93c0e29c14ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 16:48:26.359239 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:26.359203 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:26.864266 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:26.864231 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 16:48:26.864441 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:26.864291 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" podUID="69254306-affa-4f98-a1a0-93c0e29c14ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 16:48:27.356786 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.346982 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b"] Oct 30 16:48:27.364335 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.364303 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:48:27.369453 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.369430 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b"] Oct 30 16:48:27.470594 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.467938 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-72b379-serving-cert\"" Oct 30 16:48:27.470594 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.469797 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b\" (UID: \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:48:27.470594 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.469935 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjg6z\" (UniqueName: \"kubernetes.io/projected/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-kube-api-access-xjg6z\") pod \"maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b\" (UID: \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:48:27.470594 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.470012 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b\" (UID: \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:48:27.470594 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.470108 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-metrics-certs\") pod \"maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b\" (UID: \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:48:27.474863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.474805 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:27.572743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.572631 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b\" (UID: \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:48:27.572939 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.572815 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjg6z\" (UniqueName: \"kubernetes.io/projected/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-kube-api-access-xjg6z\") pod \"maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b\" (UID: \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:48:27.572939 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.572896 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b\" (UID: \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:48:27.573100 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.573000 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-metrics-certs\") pod \"maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b\" (UID: \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:48:27.585024 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.584994 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b\" (UID: \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:48:27.599841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.599669 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-metrics-certs\") pod \"maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b\" (UID: \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:48:27.615162 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.615058 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b\" (UID: \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:48:27.626284 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.626123 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:27.643491 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.643426 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjg6z\" (UniqueName: \"kubernetes.io/projected/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-kube-api-access-xjg6z\") pod \"maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b\" (UID: \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:48:27.713752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.689957 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:48:27.865408 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.865335 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Oct 30 16:48:27.865408 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:27.865385 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" podUID="69254306-affa-4f98-a1a0-93c0e29c14ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Oct 30 16:48:28.100377 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:28.100340 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:28.159977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:28.159946 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b"] Oct 30 16:48:28.167464 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:48:28.167419 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331f3eb5_061d_40f1_9ff2_9f1e90e3619c.slice/crio-510cfc537aeb231f7d2fca9c0dcedd832c4b64f52c9e932b8364bff9fc733c96 WatchSource:0}: Error finding container 510cfc537aeb231f7d2fca9c0dcedd832c4b64f52c9e932b8364bff9fc733c96: Status 404 returned error can't find the container with id 510cfc537aeb231f7d2fca9c0dcedd832c4b64f52c9e932b8364bff9fc733c96 Oct 30 16:48:28.287755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:28.285345 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:28.587540 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:28.587511 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" Oct 30 16:48:28.588295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:28.588265 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" Oct 30 16:48:28.998704 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:28.998670 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:48:29.000457 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:29.000434 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:48:29.033689 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:29.033650 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" event={"ID":"331f3eb5-061d-40f1-9ff2-9f1e90e3619c","Type":"ContainerStarted","Data":"52ba391142cb0fcf6b7c39fccdd05eda23f17c0d59fe1130dcac13a92b1c4ec4"} Oct 30 16:48:29.033899 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:29.033774 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" event={"ID":"331f3eb5-061d-40f1-9ff2-9f1e90e3619c","Type":"ContainerStarted","Data":"510cfc537aeb231f7d2fca9c0dcedd832c4b64f52c9e932b8364bff9fc733c96"} Oct 30 16:48:29.034187 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:29.034159 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:48:29.184864 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:29.184816 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" podStartSLOduration=1.7154362399999998 podStartE2EDuration="2.18480217s" totalImagesPullingTime="469.36593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:29.18115693 +0000 UTC m=+1539.523830981" watchObservedRunningTime="2026-10-30 16:48:29.18480217 +0000 UTC m=+1539.527476208" Oct 30 16:48:30.189622 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:30.189584 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:31.448497 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:31.448468 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:31.604785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:31.604705 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:32.215872 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:32.215802 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:33.133517 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.133483 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:33.282820 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.282785 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd"] Oct 30 16:48:33.288535 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.288514 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" Oct 30 16:48:33.307644 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.307619 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd"] Oct 30 16:48:33.309029 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.309003 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:33.314866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.314842 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ea8dca\"" Oct 30 16:48:33.315168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.315149 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ea8dca-dockercfg-nplbf\"" Oct 30 16:48:33.542810 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.428289 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66fbeb1-84ba-4fbf-b25c-21256006e1c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd\" (UID: \"b66fbeb1-84ba-4fbf-b25c-21256006e1c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" Oct 30 16:48:33.542810 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.532469 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq826\" (UniqueName: \"kubernetes.io/projected/b66fbeb1-84ba-4fbf-b25c-21256006e1c9-kube-api-access-mq826\") pod \"payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd\" (UID: \"b66fbeb1-84ba-4fbf-b25c-21256006e1c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" Oct 30 16:48:33.584343 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.580782 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk"] Oct 30 16:48:33.592043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.592011 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" Oct 30 16:48:33.594010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.593230 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk"] Oct 30 16:48:33.633985 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.633955 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq826\" (UniqueName: \"kubernetes.io/projected/b66fbeb1-84ba-4fbf-b25c-21256006e1c9-kube-api-access-mq826\") pod \"payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd\" (UID: \"b66fbeb1-84ba-4fbf-b25c-21256006e1c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" Oct 30 16:48:33.634126 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.634022 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ece6d8-cd04-4890-a3e2-a865b649d884-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk\" (UID: \"35ece6d8-cd04-4890-a3e2-a865b649d884\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" Oct 30 16:48:33.634126 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.634054 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66fbeb1-84ba-4fbf-b25c-21256006e1c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd\" (UID: \"b66fbeb1-84ba-4fbf-b25c-21256006e1c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" Oct 30 16:48:33.634360 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.634336 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2vrt\" (UniqueName: \"kubernetes.io/projected/35ece6d8-cd04-4890-a3e2-a865b649d884-kube-api-access-g2vrt\") pod \"payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk\" (UID: \"35ece6d8-cd04-4890-a3e2-a865b649d884\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" Oct 30 16:48:33.634693 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.634672 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66fbeb1-84ba-4fbf-b25c-21256006e1c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd\" (UID: \"b66fbeb1-84ba-4fbf-b25c-21256006e1c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" Oct 30 16:48:33.659243 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.659192 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq826\" (UniqueName: \"kubernetes.io/projected/b66fbeb1-84ba-4fbf-b25c-21256006e1c9-kube-api-access-mq826\") pod \"payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd\" (UID: \"b66fbeb1-84ba-4fbf-b25c-21256006e1c9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" Oct 30 16:48:33.735297 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.735263 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2vrt\" (UniqueName: \"kubernetes.io/projected/35ece6d8-cd04-4890-a3e2-a865b649d884-kube-api-access-g2vrt\") pod \"payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk\" (UID: \"35ece6d8-cd04-4890-a3e2-a865b649d884\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" Oct 30 16:48:33.735482 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.735355 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ece6d8-cd04-4890-a3e2-a865b649d884-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk\" (UID: \"35ece6d8-cd04-4890-a3e2-a865b649d884\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" Oct 30 16:48:33.736127 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.736100 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ece6d8-cd04-4890-a3e2-a865b649d884-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk\" (UID: \"35ece6d8-cd04-4890-a3e2-a865b649d884\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" Oct 30 16:48:33.759054 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.759031 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2vrt\" (UniqueName: \"kubernetes.io/projected/35ece6d8-cd04-4890-a3e2-a865b649d884-kube-api-access-g2vrt\") pod \"payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk\" (UID: \"35ece6d8-cd04-4890-a3e2-a865b649d884\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" Oct 30 16:48:33.910957 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.910876 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" Oct 30 16:48:33.911129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:33.911039 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" Oct 30 16:48:34.242841 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:48:34.242809 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66fbeb1_84ba_4fbf_b25c_21256006e1c9.slice/crio-db0919b2177163fd2373599ba723383b0140d8afb5ab80be6a1e8f92d5e652a6 WatchSource:0}: Error finding container db0919b2177163fd2373599ba723383b0140d8afb5ab80be6a1e8f92d5e652a6: Status 404 returned error can't find the container with id db0919b2177163fd2373599ba723383b0140d8afb5ab80be6a1e8f92d5e652a6 Oct 30 16:48:34.243574 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:34.243548 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd"] Oct 30 16:48:34.257470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:34.257427 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk"] Oct 30 16:48:34.264249 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:48:34.264224 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ece6d8_cd04_4890_a3e2_a865b649d884.slice/crio-43100f160f8c62de7ede7af7b19cac170a73c11daae8b81352183ede373e8597 WatchSource:0}: Error finding container 43100f160f8c62de7ede7af7b19cac170a73c11daae8b81352183ede373e8597: Status 404 returned error can't find the container with id 43100f160f8c62de7ede7af7b19cac170a73c11daae8b81352183ede373e8597 Oct 30 16:48:34.915872 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:34.915705 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:35.047648 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:35.047616 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:48:35.092580 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:35.092545 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" event={"ID":"b66fbeb1-84ba-4fbf-b25c-21256006e1c9","Type":"ContainerStarted","Data":"48e52a80601e41ec6a609f211eeba9d7ef31d15a1dd30c2a27ff87e2d987026f"} Oct 30 16:48:35.092813 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:35.092614 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" Oct 30 16:48:35.092813 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:35.092645 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" event={"ID":"b66fbeb1-84ba-4fbf-b25c-21256006e1c9","Type":"ContainerStarted","Data":"db0919b2177163fd2373599ba723383b0140d8afb5ab80be6a1e8f92d5e652a6"} Oct 30 16:48:35.094420 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:35.094395 4861 generic.go:356] "Generic (PLEG): container finished" podID="b5f3e0b2-750b-4545-9c66-dfc005023077" containerID="c9659376aa2a0d6d014d9874d8e0be3857b0bbca6f16bfa835b338e7ea6e87ec" exitCode=0 Oct 30 16:48:35.094537 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:35.094460 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" event={"ID":"b5f3e0b2-750b-4545-9c66-dfc005023077","Type":"ContainerDied","Data":"c9659376aa2a0d6d014d9874d8e0be3857b0bbca6f16bfa835b338e7ea6e87ec"} Oct 30 16:48:35.094945 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:35.094923 4861 scope.go:122] "RemoveContainer" containerID="c9659376aa2a0d6d014d9874d8e0be3857b0bbca6f16bfa835b338e7ea6e87ec" Oct 30 16:48:35.097363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:35.096916 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:48:35.097363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:35.096997 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:48:35.097363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:35.097033 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:48:35.097607 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:35.097504 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" event={"ID":"35ece6d8-cd04-4890-a3e2-a865b649d884","Type":"ContainerStarted","Data":"80a735db863d915efaa362fa0ade3fca3d15c30d54a50ebee065c320e1b16936"} Oct 30 16:48:35.097607 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:35.097547 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" event={"ID":"35ece6d8-cd04-4890-a3e2-a865b649d884","Type":"ContainerStarted","Data":"43100f160f8c62de7ede7af7b19cac170a73c11daae8b81352183ede373e8597"} Oct 30 16:48:35.098225 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:35.097924 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" Oct 30 16:48:35.136569 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:35.136517 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" podStartSLOduration=2.136497251 podStartE2EDuration="2.136497251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:35.13429751 +0000 UTC m=+1545.476971549" watchObservedRunningTime="2026-10-30 16:48:35.136497251 +0000 UTC m=+1545.479171289" Oct 30 16:48:35.227252 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:35.226773 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" podStartSLOduration=2.226757206 podStartE2EDuration="2.226757206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:48:35.224552984 +0000 UTC m=+1545.567227021" watchObservedRunningTime="2026-10-30 16:48:35.226757206 +0000 UTC m=+1545.569431243" Oct 30 16:48:36.130380 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:36.130338 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" event={"ID":"b5f3e0b2-750b-4545-9c66-dfc005023077","Type":"ContainerStarted","Data":"d49e68ec3dd04ebdb6f6e4a371d1e9010808c485cd46f45907055d0c95fc5687"} Oct 30 16:48:36.469074 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:36.469040 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:48:36.470071 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:36.470044 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 16:48:36.470193 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:36.470096 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" podUID="b5f3e0b2-750b-4545-9c66-dfc005023077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 16:48:36.848798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:36.841954 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:37.469271 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:37.469240 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 16:48:37.469663 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:37.469291 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" podUID="b5f3e0b2-750b-4545-9c66-dfc005023077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 16:48:38.469209 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:38.469169 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 16:48:38.469417 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:38.469222 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" podUID="b5f3e0b2-750b-4545-9c66-dfc005023077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 16:48:38.827293 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:38.827197 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:38.971777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:38.966409 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:39.163072 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:39.162970 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:48:39.468824 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:39.468795 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 16:48:39.468984 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:39.468859 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" podUID="b5f3e0b2-750b-4545-9c66-dfc005023077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 16:48:40.469690 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:40.469650 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 16:48:40.470108 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:40.469715 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" podUID="b5f3e0b2-750b-4545-9c66-dfc005023077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 16:48:40.970263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:40.970234 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:41.469366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:41.469337 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 16:48:41.469528 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:41.469398 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" podUID="b5f3e0b2-750b-4545-9c66-dfc005023077" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 16:48:42.474350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:42.474320 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:48:42.475558 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:42.475535 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:48:42.799532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:42.799452 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:43.777841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:43.777805 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:45.103758 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:45.103690 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:46.133739 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:46.133698 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" Oct 30 16:48:46.907750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:46.906093 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:47.141512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:47.141484 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" Oct 30 16:48:48.653023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:48.652984 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:48.822771 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:48.821335 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:48.958653 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:48.958578 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:50.328893 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:48:50.328839 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:48:50.515990 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:50.515945 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:51.770551 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:51.770522 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:51.915055 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:51.915024 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:52.293108 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:52.293072 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:54.089105 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:54.089076 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:55.935885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:55.935851 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:57.985020 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:57.984973 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:58.130347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:58.130305 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:48:58.278133 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:48:58.278043 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:00.294037 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:00.294000 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:00.940142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:00.940110 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:02.101081 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:02.101032 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:02.547306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:02.546095 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:02.784676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:02.784638 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:03.087910 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:03.087847 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:05.023409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:05.023304 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:05.464936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:05.464853 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:49:05.464936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:05.464923 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" podUID="9db4fb72-7b88-4237-8c83-38f14f803caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:49:06.727746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:06.724564 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:06.887764 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:06.887717 4861 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" limit=1024 Oct 30 16:49:08.333156 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.333127 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr_fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc/payload-processing/0.log" Oct 30 16:49:08.547706 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.547674 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gp6rg_a2319b60-ec05-4f37-b97a-79de3c482d58/payload-processing/0.log" Oct 30 16:49:08.557490 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.557462 4861 generic.go:356] "Generic (PLEG): container finished" podID="9db4fb72-7b88-4237-8c83-38f14f803caa" containerID="1dfbad26e9dff50ec551681a222dccd3b0708f50681f62c23176ca48295f4457" exitCode=0 Oct 30 16:49:08.557673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.557541 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" event={"ID":"9db4fb72-7b88-4237-8c83-38f14f803caa","Type":"ContainerDied","Data":"1dfbad26e9dff50ec551681a222dccd3b0708f50681f62c23176ca48295f4457"} Oct 30 16:49:08.557998 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.557981 4861 scope.go:122] "RemoveContainer" containerID="1dfbad26e9dff50ec551681a222dccd3b0708f50681f62c23176ca48295f4457" Oct 30 16:49:08.559622 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.559598 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:49:08.559695 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.559665 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:49:08.559742 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.559698 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:49:08.645832 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.645802 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt"] Oct 30 16:49:08.660922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.660880 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" Oct 30 16:49:08.683496 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.682599 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-c408f1-dockercfg-p59cf\"" Oct 30 16:49:08.683496 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.682658 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-c408f1\"" Oct 30 16:49:08.718361 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.718327 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt"] Oct 30 16:49:08.742473 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.742436 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v"] Oct 30 16:49:08.762640 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.762600 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v"] Oct 30 16:49:08.762863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.762754 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" Oct 30 16:49:08.776398 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.776272 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gp6rg_a2319b60-ec05-4f37-b97a-79de3c482d58/payload-processing/0.log" Oct 30 16:49:08.810880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.810524 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d93220-40d7-4b64-ba7b-9764ad028511-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt\" (UID: \"63d93220-40d7-4b64-ba7b-9764ad028511\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" Oct 30 16:49:08.810880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.810581 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9w9l\" (UniqueName: \"kubernetes.io/projected/63d93220-40d7-4b64-ba7b-9764ad028511-kube-api-access-q9w9l\") pod \"payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt\" (UID: \"63d93220-40d7-4b64-ba7b-9764ad028511\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" Oct 30 16:49:08.976845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.911826 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p6gj\" (UniqueName: \"kubernetes.io/projected/1e318052-ec4f-406a-97ff-a1a99b698595-kube-api-access-4p6gj\") pod \"payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v\" (UID: \"1e318052-ec4f-406a-97ff-a1a99b698595\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" Oct 30 16:49:08.976845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.911922 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d93220-40d7-4b64-ba7b-9764ad028511-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt\" (UID: \"63d93220-40d7-4b64-ba7b-9764ad028511\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" Oct 30 16:49:08.976845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.911960 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9w9l\" (UniqueName: \"kubernetes.io/projected/63d93220-40d7-4b64-ba7b-9764ad028511-kube-api-access-q9w9l\") pod \"payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt\" (UID: \"63d93220-40d7-4b64-ba7b-9764ad028511\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" Oct 30 16:49:08.976845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.912050 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e318052-ec4f-406a-97ff-a1a99b698595-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v\" (UID: \"1e318052-ec4f-406a-97ff-a1a99b698595\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" Oct 30 16:49:08.976845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.913550 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d93220-40d7-4b64-ba7b-9764ad028511-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt\" (UID: \"63d93220-40d7-4b64-ba7b-9764ad028511\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" Oct 30 16:49:08.981768 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:08.981685 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz_1916010e-81c5-4927-af79-d4d9b343ace4/payload-processing/0.log" Oct 30 16:49:09.003396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.000372 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9w9l\" (UniqueName: \"kubernetes.io/projected/63d93220-40d7-4b64-ba7b-9764ad028511-kube-api-access-q9w9l\") pod \"payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt\" (UID: \"63d93220-40d7-4b64-ba7b-9764ad028511\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" Oct 30 16:49:09.014892 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.013717 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p6gj\" (UniqueName: \"kubernetes.io/projected/1e318052-ec4f-406a-97ff-a1a99b698595-kube-api-access-4p6gj\") pod \"payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v\" (UID: \"1e318052-ec4f-406a-97ff-a1a99b698595\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" Oct 30 16:49:09.014892 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.013892 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e318052-ec4f-406a-97ff-a1a99b698595-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v\" (UID: \"1e318052-ec4f-406a-97ff-a1a99b698595\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" Oct 30 16:49:09.014892 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.014835 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e318052-ec4f-406a-97ff-a1a99b698595-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v\" (UID: \"1e318052-ec4f-406a-97ff-a1a99b698595\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" Oct 30 16:49:09.022691 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.022662 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" Oct 30 16:49:09.039315 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.039284 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p6gj\" (UniqueName: \"kubernetes.io/projected/1e318052-ec4f-406a-97ff-a1a99b698595-kube-api-access-4p6gj\") pod \"payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v\" (UID: \"1e318052-ec4f-406a-97ff-a1a99b698595\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" Oct 30 16:49:09.085273 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.084185 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" Oct 30 16:49:09.166020 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.165928 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz_1916010e-81c5-4927-af79-d4d9b343ace4/payload-processing/0.log" Oct 30 16:49:09.340547 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.340460 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt"] Oct 30 16:49:09.345081 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:49:09.345027 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d93220_40d7_4b64_ba7b_9764ad028511.slice/crio-784251da3b8ca73f463db325a8c21fa9bce05c3ec0a8b4e571a0f329e41e62d3 WatchSource:0}: Error finding container 784251da3b8ca73f463db325a8c21fa9bce05c3ec0a8b4e571a0f329e41e62d3: Status 404 returned error can't find the container with id 784251da3b8ca73f463db325a8c21fa9bce05c3ec0a8b4e571a0f329e41e62d3 Oct 30 16:49:09.454436 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.450930 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v"] Oct 30 16:49:09.460090 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:49:09.460058 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e318052_ec4f_406a_97ff_a1a99b698595.slice/crio-2b76353949d3a947677ecd6c8e66bc4128994c928bc82e57a4b2107cd6850c5f WatchSource:0}: Error finding container 2b76353949d3a947677ecd6c8e66bc4128994c928bc82e57a4b2107cd6850c5f: Status 404 returned error can't find the container with id 2b76353949d3a947677ecd6c8e66bc4128994c928bc82e57a4b2107cd6850c5f Oct 30 16:49:09.567213 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.567181 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" event={"ID":"9db4fb72-7b88-4237-8c83-38f14f803caa","Type":"ContainerStarted","Data":"9ee60eb06b52bc87e0ff7ff965610e4d6f5696bd10a3b8eda36f23cae36f6ff2"} Oct 30 16:49:09.569080 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.569052 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" event={"ID":"63d93220-40d7-4b64-ba7b-9764ad028511","Type":"ContainerStarted","Data":"267bf3da7f151c1874fa513f05fa42882b7e6edbbd813558b73dd9c1b4f1baad"} Oct 30 16:49:09.569217 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.569102 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" event={"ID":"63d93220-40d7-4b64-ba7b-9764ad028511","Type":"ContainerStarted","Data":"784251da3b8ca73f463db325a8c21fa9bce05c3ec0a8b4e571a0f329e41e62d3"} Oct 30 16:49:09.569217 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.569164 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" Oct 30 16:49:09.570973 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.570947 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" event={"ID":"1e318052-ec4f-406a-97ff-a1a99b698595","Type":"ContainerStarted","Data":"aab121455eadc188746bc80841c849f20eef335f89b665e6611d7aa61bd87b17"} Oct 30 16:49:09.571067 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.570991 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" event={"ID":"1e318052-ec4f-406a-97ff-a1a99b698595","Type":"ContainerStarted","Data":"2b76353949d3a947677ecd6c8e66bc4128994c928bc82e57a4b2107cd6850c5f"} Oct 30 16:49:09.571067 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.571023 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" Oct 30 16:49:09.675513 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:09.675408 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" podStartSLOduration=1.6753926529999998 podStartE2EDuration="1.675392653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:09.672825128 +0000 UTC m=+1580.015499168" watchObservedRunningTime="2026-10-30 16:49:09.675392653 +0000 UTC m=+1580.018066717" Oct 30 16:49:10.428266 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:10.428226 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 16:49:10.429268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:10.428905 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" podUID="9db4fb72-7b88-4237-8c83-38f14f803caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 16:49:10.429268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:10.428787 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:49:11.427995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:11.427959 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 16:49:11.428294 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:11.428263 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" podUID="9db4fb72-7b88-4237-8c83-38f14f803caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 16:49:11.442098 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:11.442043 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" podStartSLOduration=3.442026709 podStartE2EDuration="3.442026709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:09.724040962 +0000 UTC m=+1580.066715001" watchObservedRunningTime="2026-10-30 16:49:11.442026709 +0000 UTC m=+1581.784700803" Oct 30 16:49:11.443235 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:11.443214 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57"] Oct 30 16:49:11.475796 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:11.475756 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg"] Oct 30 16:49:12.430009 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:12.429500 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 16:49:12.430009 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:12.429579 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" podUID="9db4fb72-7b88-4237-8c83-38f14f803caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 16:49:13.428131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:13.428053 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 16:49:13.428131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:13.428105 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" podUID="9db4fb72-7b88-4237-8c83-38f14f803caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 16:49:13.677317 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:13.677233 4861 generic.go:356] "Generic (PLEG): container finished" podID="ab025762-9af1-4c25-a0cc-4ab9c0221987" containerID="787fff535d36a91672b7d646daf82f9e95b75a9c6d6d43948e30a7472849d818" exitCode=0 Oct 30 16:49:13.677684 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:13.677400 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" event={"ID":"ab025762-9af1-4c25-a0cc-4ab9c0221987","Type":"ContainerDied","Data":"787fff535d36a91672b7d646daf82f9e95b75a9c6d6d43948e30a7472849d818"} Oct 30 16:49:13.677859 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:13.677841 4861 scope.go:122] "RemoveContainer" containerID="787fff535d36a91672b7d646daf82f9e95b75a9c6d6d43948e30a7472849d818" Oct 30 16:49:13.679760 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:13.679689 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:49:13.679869 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:13.679791 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:49:13.679869 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:13.679827 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:49:14.452754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:14.451938 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Oct 30 16:49:14.452754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:14.452003 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" podUID="9db4fb72-7b88-4237-8c83-38f14f803caa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Oct 30 16:49:14.686510 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:14.686480 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" event={"ID":"ab025762-9af1-4c25-a0cc-4ab9c0221987","Type":"ContainerStarted","Data":"145ae9daaea2719442e60284123893434d1252b076cf2620bd82f9b919872c02"} Oct 30 16:49:15.067653 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:15.067619 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:49:15.067876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:15.067841 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 16:49:15.067960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:15.067896 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" podUID="ab025762-9af1-4c25-a0cc-4ab9c0221987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 16:49:15.433035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:15.432953 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:49:15.433391 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:15.433370 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:49:15.434066 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:15.434050 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:49:16.068584 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:16.068546 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 16:49:16.068989 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:16.068606 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" podUID="ab025762-9af1-4c25-a0cc-4ab9c0221987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 16:49:16.444231 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:16.444137 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" podUID="936e6d9b-be30-4ea6-90e1-bcc3787fe99a" containerName="payload-processing" containerID="cri-o://7f3f43f199df15be0874d772ca2f29cddd4b071a8b0b8d2fbf0787aa21b33ca3" gracePeriod=25 Oct 30 16:49:16.476122 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:16.476069 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" podUID="5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2" containerName="payload-pre-processing" containerID="cri-o://1c28615d336f50d00d06ba3317c84a44953d475cd2baa7b8b64afd99795920fa" gracePeriod=25 Oct 30 16:49:16.730763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:16.726599 4861 generic.go:356] "Generic (PLEG): container finished" podID="936e6d9b-be30-4ea6-90e1-bcc3787fe99a" containerID="7f3f43f199df15be0874d772ca2f29cddd4b071a8b0b8d2fbf0787aa21b33ca3" exitCode=0 Oct 30 16:49:16.730763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:16.726802 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" event={"ID":"936e6d9b-be30-4ea6-90e1-bcc3787fe99a","Type":"ContainerDied","Data":"7f3f43f199df15be0874d772ca2f29cddd4b071a8b0b8d2fbf0787aa21b33ca3"} Oct 30 16:49:16.731779 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:16.731712 4861 generic.go:356] "Generic (PLEG): container finished" podID="5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2" containerID="1c28615d336f50d00d06ba3317c84a44953d475cd2baa7b8b64afd99795920fa" exitCode=0 Oct 30 16:49:16.731938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:16.731843 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" event={"ID":"5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2","Type":"ContainerDied","Data":"1c28615d336f50d00d06ba3317c84a44953d475cd2baa7b8b64afd99795920fa"} Oct 30 16:49:16.906761 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:16.904893 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" Oct 30 16:49:16.931749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:16.929786 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2-kube-api-access-ng5x2\" (UniqueName: \"kubernetes.io/projected/5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2-kube-api-access-ng5x2\") pod \"5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2\" (UID: \"5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2\") " Oct 30 16:49:16.931749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:16.930090 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2-plugins-config-volume\") pod \"5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2\" (UID: \"5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2\") " Oct 30 16:49:16.931749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:16.931652 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2-plugins-config-volume" pod "5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2" (UID: "5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:16.934321 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:16.934297 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2-kube-api-access-ng5x2" pod "5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2" (UID: "5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2"). InnerVolumeSpecName "kube-api-access-ng5x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:16.958056 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:16.958032 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" Oct 30 16:49:17.037200 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.037111 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:17.037200 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.037140 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng5x2\" (UniqueName: \"kubernetes.io/projected/5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2-kube-api-access-ng5x2\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:17.068860 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.068828 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 16:49:17.069264 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.068889 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" podUID="ab025762-9af1-4c25-a0cc-4ab9c0221987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 16:49:17.137668 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.137634 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936e6d9b-be30-4ea6-90e1-bcc3787fe99a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936e6d9b-be30-4ea6-90e1-bcc3787fe99a-plugins-config-volume\") pod \"936e6d9b-be30-4ea6-90e1-bcc3787fe99a\" (UID: \"936e6d9b-be30-4ea6-90e1-bcc3787fe99a\") " Oct 30 16:49:17.137880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.137844 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936e6d9b-be30-4ea6-90e1-bcc3787fe99a-kube-api-access-fcjfc\" (UniqueName: \"kubernetes.io/projected/936e6d9b-be30-4ea6-90e1-bcc3787fe99a-kube-api-access-fcjfc\") pod \"936e6d9b-be30-4ea6-90e1-bcc3787fe99a\" (UID: \"936e6d9b-be30-4ea6-90e1-bcc3787fe99a\") " Oct 30 16:49:17.138087 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.138062 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936e6d9b-be30-4ea6-90e1-bcc3787fe99a-plugins-config-volume" pod "936e6d9b-be30-4ea6-90e1-bcc3787fe99a" (UID: "936e6d9b-be30-4ea6-90e1-bcc3787fe99a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:17.138553 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.138527 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/936e6d9b-be30-4ea6-90e1-bcc3787fe99a-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:17.140575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.140552 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936e6d9b-be30-4ea6-90e1-bcc3787fe99a-kube-api-access-fcjfc" pod "936e6d9b-be30-4ea6-90e1-bcc3787fe99a" (UID: "936e6d9b-be30-4ea6-90e1-bcc3787fe99a"). InnerVolumeSpecName "kube-api-access-fcjfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:17.240145 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.240113 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcjfc\" (UniqueName: \"kubernetes.io/projected/936e6d9b-be30-4ea6-90e1-bcc3787fe99a-kube-api-access-fcjfc\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:17.742063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.742034 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" Oct 30 16:49:17.742271 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.742063 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57" event={"ID":"936e6d9b-be30-4ea6-90e1-bcc3787fe99a","Type":"ContainerDied","Data":"fc73b88d8945491caa00f381ad8ada3f88b91b4c6689b87b41ee8adce0e3ae11"} Oct 30 16:49:17.742271 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.742127 4861 scope.go:122] "RemoveContainer" containerID="7f3f43f199df15be0874d772ca2f29cddd4b071a8b0b8d2fbf0787aa21b33ca3" Oct 30 16:49:17.744224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.744195 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" event={"ID":"5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2","Type":"ContainerDied","Data":"44262f1b3afa5d7e838459f8d29b5ac474133f641369f18a9d690f07f5154109"} Oct 30 16:49:17.744331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.744239 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg" Oct 30 16:49:17.753661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.753639 4861 scope.go:122] "RemoveContainer" containerID="1c28615d336f50d00d06ba3317c84a44953d475cd2baa7b8b64afd99795920fa" Oct 30 16:49:17.798425 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.798399 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57"] Oct 30 16:49:17.846399 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.846363 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms"] Oct 30 16:49:17.847470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.847443 4861 cpu_manager.go:414] "containerMap: removing container" podUID="5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2" containerName="payload-pre-processing" Oct 30 16:49:17.847470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.847471 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2" containerName="payload-pre-processing" Oct 30 16:49:17.847671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.847503 4861 cpu_manager.go:414] "containerMap: removing container" podUID="936e6d9b-be30-4ea6-90e1-bcc3787fe99a" containerName="payload-processing" Oct 30 16:49:17.847671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.847511 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936e6d9b-be30-4ea6-90e1-bcc3787fe99a" containerName="payload-processing" Oct 30 16:49:17.848089 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.847767 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2" containerName="payload-pre-processing" Oct 30 16:49:17.848089 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.847787 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="936e6d9b-be30-4ea6-90e1-bcc3787fe99a" containerName="payload-processing" Oct 30 16:49:17.854047 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.854025 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cfe9f0-7bb9ccf674-b4r57"] Oct 30 16:49:17.854153 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.854066 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms"] Oct 30 16:49:17.854153 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.854100 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg"] Oct 30 16:49:17.854153 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.854126 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" Oct 30 16:49:17.858844 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.858819 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-cfe9f0-6bb7cb5b59-wdjkg"] Oct 30 16:49:17.871963 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.871861 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6d5903\"" Oct 30 16:49:17.872929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.872842 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6d5903-dockercfg-x7g8x\"" Oct 30 16:49:17.967673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.967623 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hdw2\" (UniqueName: \"kubernetes.io/projected/0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5-kube-api-access-6hdw2\") pod \"payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms\" (UID: \"0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" Oct 30 16:49:17.968155 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:17.968132 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms\" (UID: \"0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" Oct 30 16:49:18.108488 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.108388 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms\" (UID: \"0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" Oct 30 16:49:18.152554 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.120620 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms\" (UID: \"0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" Oct 30 16:49:18.152768 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.120664 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74"] Oct 30 16:49:18.153401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.153376 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 16:49:18.153891 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.153854 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" podUID="ab025762-9af1-4c25-a0cc-4ab9c0221987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 16:49:18.156562 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.156533 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hdw2\" (UniqueName: \"kubernetes.io/projected/0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5-kube-api-access-6hdw2\") pod \"payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms\" (UID: \"0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" Oct 30 16:49:18.159014 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.158986 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74"] Oct 30 16:49:18.159129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.159096 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" Oct 30 16:49:18.180416 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.180392 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hdw2\" (UniqueName: \"kubernetes.io/projected/0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5-kube-api-access-6hdw2\") pod \"payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms\" (UID: \"0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" Oct 30 16:49:18.250612 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.250579 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" Oct 30 16:49:18.257944 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.257921 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6th8h\" (UniqueName: \"kubernetes.io/projected/d5a2c326-4e32-4826-b24d-b9ea22ef5e3e-kube-api-access-6th8h\") pod \"payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74\" (UID: \"d5a2c326-4e32-4826-b24d-b9ea22ef5e3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" Oct 30 16:49:18.258076 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.257960 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a2c326-4e32-4826-b24d-b9ea22ef5e3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74\" (UID: \"d5a2c326-4e32-4826-b24d-b9ea22ef5e3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" Oct 30 16:49:18.359717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.359638 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6th8h\" (UniqueName: \"kubernetes.io/projected/d5a2c326-4e32-4826-b24d-b9ea22ef5e3e-kube-api-access-6th8h\") pod \"payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74\" (UID: \"d5a2c326-4e32-4826-b24d-b9ea22ef5e3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" Oct 30 16:49:18.359717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.359699 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a2c326-4e32-4826-b24d-b9ea22ef5e3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74\" (UID: \"d5a2c326-4e32-4826-b24d-b9ea22ef5e3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" Oct 30 16:49:18.360699 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.360674 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a2c326-4e32-4826-b24d-b9ea22ef5e3e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74\" (UID: \"d5a2c326-4e32-4826-b24d-b9ea22ef5e3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" Oct 30 16:49:18.385257 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.385234 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6th8h\" (UniqueName: \"kubernetes.io/projected/d5a2c326-4e32-4826-b24d-b9ea22ef5e3e-kube-api-access-6th8h\") pod \"payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74\" (UID: \"d5a2c326-4e32-4826-b24d-b9ea22ef5e3e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" Oct 30 16:49:18.478286 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.478220 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" Oct 30 16:49:18.614808 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.613862 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms"] Oct 30 16:49:18.765304 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.765273 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" event={"ID":"0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5","Type":"ContainerStarted","Data":"4a426ef953ae72196e55e69f09dd3bde313f9d881e2981466c8ea22c557fe505"} Oct 30 16:49:18.849416 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:18.849376 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74"] Oct 30 16:49:18.866483 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:49:18.866424 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a2c326_4e32_4826_b24d_b9ea22ef5e3e.slice/crio-f97fda372654ddc1adbdcc311b87f711978d23b88c4e5f9970450cd8f601fde4 WatchSource:0}: Error finding container f97fda372654ddc1adbdcc311b87f711978d23b88c4e5f9970450cd8f601fde4: Status 404 returned error can't find the container with id f97fda372654ddc1adbdcc311b87f711978d23b88c4e5f9970450cd8f601fde4 Oct 30 16:49:19.068431 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:19.068395 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 16:49:19.068592 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:19.068446 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" podUID="ab025762-9af1-4c25-a0cc-4ab9c0221987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 16:49:19.793068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:19.793029 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" event={"ID":"0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5","Type":"ContainerStarted","Data":"440a5e8ff59121692c4d6d87ad1cb41eb7c9c2c5c24b93803328df68fcda36a5"} Oct 30 16:49:19.793545 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:19.793090 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" Oct 30 16:49:19.794954 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:19.794930 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" event={"ID":"d5a2c326-4e32-4826-b24d-b9ea22ef5e3e","Type":"ContainerStarted","Data":"64b4f519412477e1d2a91995c7f643b872f67124320431c5a92f64a4c1c89341"} Oct 30 16:49:19.795100 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:19.794972 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" event={"ID":"d5a2c326-4e32-4826-b24d-b9ea22ef5e3e","Type":"ContainerStarted","Data":"f97fda372654ddc1adbdcc311b87f711978d23b88c4e5f9970450cd8f601fde4"} Oct 30 16:49:19.795100 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:19.794994 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" Oct 30 16:49:19.834846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:19.834819 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2" path="/var/lib/kubelet/pods/5a895fbb-1e0e-4c0d-bf08-48de9c92e1f2/volumes" Oct 30 16:49:19.835293 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:19.835279 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936e6d9b-be30-4ea6-90e1-bcc3787fe99a" path="/var/lib/kubelet/pods/936e6d9b-be30-4ea6-90e1-bcc3787fe99a/volumes" Oct 30 16:49:19.853195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:19.853150 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" podStartSLOduration=2.8531377300000003 podStartE2EDuration="2.85313773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:19.85023769 +0000 UTC m=+1590.192911729" watchObservedRunningTime="2026-10-30 16:49:19.85313773 +0000 UTC m=+1590.195811768" Oct 30 16:49:19.922160 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:19.922105 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" podStartSLOduration=2.922091854 podStartE2EDuration="2.922091854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:19.919386459 +0000 UTC m=+1590.262060499" watchObservedRunningTime="2026-10-30 16:49:19.922091854 +0000 UTC m=+1590.264765891" Oct 30 16:49:20.077798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:20.069528 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Oct 30 16:49:20.077798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:20.069606 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" podUID="ab025762-9af1-4c25-a0cc-4ab9c0221987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Oct 30 16:49:20.290638 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:49:20.290603 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:49:20.597713 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:20.597684 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" Oct 30 16:49:20.598520 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:20.598494 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" Oct 30 16:49:20.653942 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:20.653905 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:49:21.072090 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:21.072055 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:49:21.073172 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:21.073148 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:49:30.137612 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:30.137577 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj"] Oct 30 16:49:30.138203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:30.137852 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" podUID="12ea7c70-5553-4086-b4f5-dc5895c4f0a1" containerName="istio-proxy" containerID="cri-o://2c63f603b2ce4f499af7a5b1ce96f86258fbe4902266e146b342284a67fc39b1" gracePeriod=30 Oct 30 16:49:30.805498 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:30.805470 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" Oct 30 16:49:30.805734 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:30.805541 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" Oct 30 16:49:33.850931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:33.850900 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl"] Oct 30 16:49:33.857013 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:33.856993 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:49:33.872601 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:33.872568 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-c408f1-serving-cert\"" Oct 30 16:49:33.874207 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:33.874185 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl"] Oct 30 16:49:33.905021 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:33.904985 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac9dbe1-f2bc-4feb-b107-b94cca178634-metrics-certs\") pod \"maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl\" (UID: \"dac9dbe1-f2bc-4feb-b107-b94cca178634\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:49:33.905165 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:33.905025 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgnvl\" (UniqueName: \"kubernetes.io/projected/dac9dbe1-f2bc-4feb-b107-b94cca178634-kube-api-access-qgnvl\") pod \"maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl\" (UID: \"dac9dbe1-f2bc-4feb-b107-b94cca178634\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:49:33.905216 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:33.905188 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac9dbe1-f2bc-4feb-b107-b94cca178634-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl\" (UID: \"dac9dbe1-f2bc-4feb-b107-b94cca178634\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:49:33.905301 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:33.905275 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dac9dbe1-f2bc-4feb-b107-b94cca178634-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl\" (UID: \"dac9dbe1-f2bc-4feb-b107-b94cca178634\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:49:34.006894 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:34.006863 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac9dbe1-f2bc-4feb-b107-b94cca178634-metrics-certs\") pod \"maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl\" (UID: \"dac9dbe1-f2bc-4feb-b107-b94cca178634\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:49:34.006894 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:34.006898 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgnvl\" (UniqueName: \"kubernetes.io/projected/dac9dbe1-f2bc-4feb-b107-b94cca178634-kube-api-access-qgnvl\") pod \"maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl\" (UID: \"dac9dbe1-f2bc-4feb-b107-b94cca178634\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:49:34.007189 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:34.006966 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac9dbe1-f2bc-4feb-b107-b94cca178634-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl\" (UID: \"dac9dbe1-f2bc-4feb-b107-b94cca178634\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:49:34.007189 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:34.007015 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dac9dbe1-f2bc-4feb-b107-b94cca178634-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl\" (UID: \"dac9dbe1-f2bc-4feb-b107-b94cca178634\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:49:34.007818 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:34.007715 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac9dbe1-f2bc-4feb-b107-b94cca178634-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl\" (UID: \"dac9dbe1-f2bc-4feb-b107-b94cca178634\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:49:34.009577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:34.009544 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dac9dbe1-f2bc-4feb-b107-b94cca178634-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl\" (UID: \"dac9dbe1-f2bc-4feb-b107-b94cca178634\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:49:34.009691 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:34.009667 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac9dbe1-f2bc-4feb-b107-b94cca178634-metrics-certs\") pod \"maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl\" (UID: \"dac9dbe1-f2bc-4feb-b107-b94cca178634\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:49:34.031106 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:34.031080 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgnvl\" (UniqueName: \"kubernetes.io/projected/dac9dbe1-f2bc-4feb-b107-b94cca178634-kube-api-access-qgnvl\") pod \"maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl\" (UID: \"dac9dbe1-f2bc-4feb-b107-b94cca178634\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:49:34.180503 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:34.176755 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:49:34.491645 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:34.491615 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl"] Oct 30 16:49:34.492332 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:49:34.492306 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac9dbe1_f2bc_4feb_b107_b94cca178634.slice/crio-595611a5e22de2200a3fbe12e012f69d1be8439555e0872f14a47d5141f87afa WatchSource:0}: Error finding container 595611a5e22de2200a3fbe12e012f69d1be8439555e0872f14a47d5141f87afa: Status 404 returned error can't find the container with id 595611a5e22de2200a3fbe12e012f69d1be8439555e0872f14a47d5141f87afa Oct 30 16:49:35.000623 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.000576 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" event={"ID":"dac9dbe1-f2bc-4feb-b107-b94cca178634","Type":"ContainerStarted","Data":"595611a5e22de2200a3fbe12e012f69d1be8439555e0872f14a47d5141f87afa"} Oct 30 16:49:35.555151 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.555125 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:49:35.662205 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.662114 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-podinfo\") pod \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " Oct 30 16:49:35.662391 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.662229 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-workload-socket\") pod \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " Oct 30 16:49:35.662391 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.662248 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-envoy\") pod \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " Oct 30 16:49:35.662391 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.662325 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-workload-certs\") pod \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " Oct 30 16:49:35.662572 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.662402 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-credential-socket\") pod \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " Oct 30 16:49:35.662572 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.662436 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-token\" (UniqueName: \"kubernetes.io/projected/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-token\") pod \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " Oct 30 16:49:35.662572 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.662551 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istiod-ca-cert\") pod \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " Oct 30 16:49:35.662755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.662579 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-data\") pod \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " Oct 30 16:49:35.662755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.662617 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-kube-api-access-c4lzj\" (UniqueName: \"kubernetes.io/projected/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-kube-api-access-c4lzj\") pod \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\" (UID: \"12ea7c70-5553-4086-b4f5-dc5895c4f0a1\") " Oct 30 16:49:35.662755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.662700 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-credential-socket" pod "12ea7c70-5553-4086-b4f5-dc5895c4f0a1" (UID: "12ea7c70-5553-4086-b4f5-dc5895c4f0a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.663330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.662979 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-workload-socket" pod "12ea7c70-5553-4086-b4f5-dc5895c4f0a1" (UID: "12ea7c70-5553-4086-b4f5-dc5895c4f0a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.663330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.663206 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-data" pod "12ea7c70-5553-4086-b4f5-dc5895c4f0a1" (UID: "12ea7c70-5553-4086-b4f5-dc5895c4f0a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.663330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.663216 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istiod-ca-cert" pod "12ea7c70-5553-4086-b4f5-dc5895c4f0a1" (UID: "12ea7c70-5553-4086-b4f5-dc5895c4f0a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:35.663533 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.663507 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.663595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.663543 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.663595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.663559 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.663595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.663574 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.663796 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.663770 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-workload-certs" pod "12ea7c70-5553-4086-b4f5-dc5895c4f0a1" (UID: "12ea7c70-5553-4086-b4f5-dc5895c4f0a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.664835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.664809 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-podinfo" pod "12ea7c70-5553-4086-b4f5-dc5895c4f0a1" (UID: "12ea7c70-5553-4086-b4f5-dc5895c4f0a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:49:35.665111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.665060 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-envoy" pod "12ea7c70-5553-4086-b4f5-dc5895c4f0a1" (UID: "12ea7c70-5553-4086-b4f5-dc5895c4f0a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:49:35.665819 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.665797 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-token" pod "12ea7c70-5553-4086-b4f5-dc5895c4f0a1" (UID: "12ea7c70-5553-4086-b4f5-dc5895c4f0a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:35.666026 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.666009 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-kube-api-access-c4lzj" pod "12ea7c70-5553-4086-b4f5-dc5895c4f0a1" (UID: "12ea7c70-5553-4086-b4f5-dc5895c4f0a1"). InnerVolumeSpecName "kube-api-access-c4lzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:35.770750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.769875 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.770750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.769949 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.770750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.769987 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4lzj\" (UniqueName: \"kubernetes.io/projected/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-kube-api-access-c4lzj\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.770750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.770004 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:35.770750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:35.770019 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12ea7c70-5553-4086-b4f5-dc5895c4f0a1-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:36.011395 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:36.011353 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" event={"ID":"dac9dbe1-f2bc-4feb-b107-b94cca178634","Type":"ContainerStarted","Data":"a66ae9b7e12745f797b6d15043b8d1a1a28ae50ab5f6a2211f6a17c6cca26cfa"} Oct 30 16:49:36.011936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:36.011916 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:49:36.013990 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:36.013954 4861 generic.go:356] "Generic (PLEG): container finished" podID="12ea7c70-5553-4086-b4f5-dc5895c4f0a1" containerID="2c63f603b2ce4f499af7a5b1ce96f86258fbe4902266e146b342284a67fc39b1" exitCode=0 Oct 30 16:49:36.014118 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:36.014000 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" Oct 30 16:49:36.014118 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:36.014027 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" event={"ID":"12ea7c70-5553-4086-b4f5-dc5895c4f0a1","Type":"ContainerDied","Data":"2c63f603b2ce4f499af7a5b1ce96f86258fbe4902266e146b342284a67fc39b1"} Oct 30 16:49:36.014118 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:36.014093 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj" event={"ID":"12ea7c70-5553-4086-b4f5-dc5895c4f0a1","Type":"ContainerDied","Data":"d5d5f5d6bb075edc869152c162cd9adb9d4d683b8bba0658e44610487f45807c"} Oct 30 16:49:36.014288 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:36.014126 4861 scope.go:122] "RemoveContainer" containerID="2c63f603b2ce4f499af7a5b1ce96f86258fbe4902266e146b342284a67fc39b1" Oct 30 16:49:36.035733 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:36.035704 4861 scope.go:122] "RemoveContainer" containerID="176aa82fc35ddbc16b790d118a27eb080fd0d796c4c2eb5045b36c9eb3b63a5f" Oct 30 16:49:36.046580 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:36.046522 4861 scope.go:122] "RemoveContainer" containerID="2c63f603b2ce4f499af7a5b1ce96f86258fbe4902266e146b342284a67fc39b1" Oct 30 16:49:36.046910 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:49:36.046880 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c63f603b2ce4f499af7a5b1ce96f86258fbe4902266e146b342284a67fc39b1\": container with ID starting with 2c63f603b2ce4f499af7a5b1ce96f86258fbe4902266e146b342284a67fc39b1 not found: ID does not exist" containerID="2c63f603b2ce4f499af7a5b1ce96f86258fbe4902266e146b342284a67fc39b1" Oct 30 16:49:36.047044 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:36.046918 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c63f603b2ce4f499af7a5b1ce96f86258fbe4902266e146b342284a67fc39b1"} err="failed to get container status \"2c63f603b2ce4f499af7a5b1ce96f86258fbe4902266e146b342284a67fc39b1\": rpc error: code = NotFound desc = could not find container \"2c63f603b2ce4f499af7a5b1ce96f86258fbe4902266e146b342284a67fc39b1\": container with ID starting with 2c63f603b2ce4f499af7a5b1ce96f86258fbe4902266e146b342284a67fc39b1 not found: ID does not exist" Oct 30 16:49:36.047044 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:36.046943 4861 scope.go:122] "RemoveContainer" containerID="176aa82fc35ddbc16b790d118a27eb080fd0d796c4c2eb5045b36c9eb3b63a5f" Oct 30 16:49:36.047267 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:49:36.047243 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"176aa82fc35ddbc16b790d118a27eb080fd0d796c4c2eb5045b36c9eb3b63a5f\": container with ID starting with 176aa82fc35ddbc16b790d118a27eb080fd0d796c4c2eb5045b36c9eb3b63a5f not found: ID does not exist" containerID="176aa82fc35ddbc16b790d118a27eb080fd0d796c4c2eb5045b36c9eb3b63a5f" Oct 30 16:49:36.047337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:36.047274 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"176aa82fc35ddbc16b790d118a27eb080fd0d796c4c2eb5045b36c9eb3b63a5f"} err="failed to get container status \"176aa82fc35ddbc16b790d118a27eb080fd0d796c4c2eb5045b36c9eb3b63a5f\": rpc error: code = NotFound desc = could not find container \"176aa82fc35ddbc16b790d118a27eb080fd0d796c4c2eb5045b36c9eb3b63a5f\": container with ID starting with 176aa82fc35ddbc16b790d118a27eb080fd0d796c4c2eb5045b36c9eb3b63a5f not found: ID does not exist" Oct 30 16:49:36.059544 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:36.059490 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" podStartSLOduration=2.507722793 podStartE2EDuration="3.059472594s" totalImagesPullingTime="551.749801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:49:36.056981423 +0000 UTC m=+1606.399655460" watchObservedRunningTime="2026-10-30 16:49:36.059472594 +0000 UTC m=+1606.402146636" Oct 30 16:49:36.096474 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:36.096441 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj"] Oct 30 16:49:36.109049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:36.109018 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2qsbj"] Oct 30 16:49:37.833880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:37.833846 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12ea7c70-5553-4086-b4f5-dc5895c4f0a1" path="/var/lib/kubelet/pods/12ea7c70-5553-4086-b4f5-dc5895c4f0a1/volumes" Oct 30 16:49:41.043118 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:41.043083 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf"] Oct 30 16:49:41.078040 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:41.077950 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54"] Oct 30 16:49:42.029029 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:42.028999 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:49:45.303387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:45.303351 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z"] Oct 30 16:49:45.341267 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:45.341238 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz"] Oct 30 16:49:46.050592 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.044813 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" podUID="c35fb6ac-30a4-489b-827e-90831f6827d8" containerName="payload-processing" containerID="cri-o://4c1e43473b8348da52eb260f72e3da580b724b460143702d7fa04250968de64d" gracePeriod=25 Oct 30 16:49:46.080662 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.079800 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" podUID="824cd0d9-5af5-4c68-a4c8-62ff61852acc" containerName="payload-pre-processing" containerID="cri-o://3be7e0d7828064286a3cbdcc9ae51a416a5e440487ba04cf8aaace557b54709b" gracePeriod=25 Oct 30 16:49:46.198532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.198206 4861 generic.go:356] "Generic (PLEG): container finished" podID="c35fb6ac-30a4-489b-827e-90831f6827d8" containerID="4c1e43473b8348da52eb260f72e3da580b724b460143702d7fa04250968de64d" exitCode=0 Oct 30 16:49:46.198532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.198375 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" event={"ID":"c35fb6ac-30a4-489b-827e-90831f6827d8","Type":"ContainerDied","Data":"4c1e43473b8348da52eb260f72e3da580b724b460143702d7fa04250968de64d"} Oct 30 16:49:46.410857 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.410827 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" Oct 30 16:49:46.511767 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.511692 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" Oct 30 16:49:46.538515 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.538488 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c35fb6ac-30a4-489b-827e-90831f6827d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35fb6ac-30a4-489b-827e-90831f6827d8-plugins-config-volume\") pod \"c35fb6ac-30a4-489b-827e-90831f6827d8\" (UID: \"c35fb6ac-30a4-489b-827e-90831f6827d8\") " Oct 30 16:49:46.538686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.538588 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c35fb6ac-30a4-489b-827e-90831f6827d8-kube-api-access-6xlsw\" (UniqueName: \"kubernetes.io/projected/c35fb6ac-30a4-489b-827e-90831f6827d8-kube-api-access-6xlsw\") pod \"c35fb6ac-30a4-489b-827e-90831f6827d8\" (UID: \"c35fb6ac-30a4-489b-827e-90831f6827d8\") " Oct 30 16:49:46.538863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.538842 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35fb6ac-30a4-489b-827e-90831f6827d8-plugins-config-volume" pod "c35fb6ac-30a4-489b-827e-90831f6827d8" (UID: "c35fb6ac-30a4-489b-827e-90831f6827d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:46.539254 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.539235 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35fb6ac-30a4-489b-827e-90831f6827d8-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:46.540984 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.540963 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35fb6ac-30a4-489b-827e-90831f6827d8-kube-api-access-6xlsw" pod "c35fb6ac-30a4-489b-827e-90831f6827d8" (UID: "c35fb6ac-30a4-489b-827e-90831f6827d8"). InnerVolumeSpecName "kube-api-access-6xlsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:46.639683 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.639602 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824cd0d9-5af5-4c68-a4c8-62ff61852acc-kube-api-access-rczrd\" (UniqueName: \"kubernetes.io/projected/824cd0d9-5af5-4c68-a4c8-62ff61852acc-kube-api-access-rczrd\") pod \"824cd0d9-5af5-4c68-a4c8-62ff61852acc\" (UID: \"824cd0d9-5af5-4c68-a4c8-62ff61852acc\") " Oct 30 16:49:46.639683 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.639657 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824cd0d9-5af5-4c68-a4c8-62ff61852acc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824cd0d9-5af5-4c68-a4c8-62ff61852acc-plugins-config-volume\") pod \"824cd0d9-5af5-4c68-a4c8-62ff61852acc\" (UID: \"824cd0d9-5af5-4c68-a4c8-62ff61852acc\") " Oct 30 16:49:46.640070 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.640049 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824cd0d9-5af5-4c68-a4c8-62ff61852acc-plugins-config-volume" pod "824cd0d9-5af5-4c68-a4c8-62ff61852acc" (UID: "824cd0d9-5af5-4c68-a4c8-62ff61852acc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:46.640280 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.640264 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824cd0d9-5af5-4c68-a4c8-62ff61852acc-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:46.640321 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.640284 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xlsw\" (UniqueName: \"kubernetes.io/projected/c35fb6ac-30a4-489b-827e-90831f6827d8-kube-api-access-6xlsw\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:46.641926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.641902 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824cd0d9-5af5-4c68-a4c8-62ff61852acc-kube-api-access-rczrd" pod "824cd0d9-5af5-4c68-a4c8-62ff61852acc" (UID: "824cd0d9-5af5-4c68-a4c8-62ff61852acc"). InnerVolumeSpecName "kube-api-access-rczrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:46.742094 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:46.742067 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rczrd\" (UniqueName: \"kubernetes.io/projected/824cd0d9-5af5-4c68-a4c8-62ff61852acc-kube-api-access-rczrd\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:47.206859 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:47.206821 4861 generic.go:356] "Generic (PLEG): container finished" podID="824cd0d9-5af5-4c68-a4c8-62ff61852acc" containerID="3be7e0d7828064286a3cbdcc9ae51a416a5e440487ba04cf8aaace557b54709b" exitCode=0 Oct 30 16:49:47.207063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:47.206874 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" Oct 30 16:49:47.207063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:47.206912 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" event={"ID":"824cd0d9-5af5-4c68-a4c8-62ff61852acc","Type":"ContainerDied","Data":"3be7e0d7828064286a3cbdcc9ae51a416a5e440487ba04cf8aaace557b54709b"} Oct 30 16:49:47.207063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:47.206969 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54" event={"ID":"824cd0d9-5af5-4c68-a4c8-62ff61852acc","Type":"ContainerDied","Data":"17978c4aa8af90bf6d5d838433ff2842de4294b0f68d04d1d5447f8c0f510510"} Oct 30 16:49:47.207063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:47.206997 4861 scope.go:122] "RemoveContainer" containerID="3be7e0d7828064286a3cbdcc9ae51a416a5e440487ba04cf8aaace557b54709b" Oct 30 16:49:47.208586 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:47.208569 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" Oct 30 16:49:47.208712 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:47.208600 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf" event={"ID":"c35fb6ac-30a4-489b-827e-90831f6827d8","Type":"ContainerDied","Data":"ed5b30650fe317c3a558ca787b1aea7639d70adae7926763a4f856a55046e6cf"} Oct 30 16:49:47.218415 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:47.218312 4861 scope.go:122] "RemoveContainer" containerID="3be7e0d7828064286a3cbdcc9ae51a416a5e440487ba04cf8aaace557b54709b" Oct 30 16:49:47.218595 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:49:47.218567 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3be7e0d7828064286a3cbdcc9ae51a416a5e440487ba04cf8aaace557b54709b\": container with ID starting with 3be7e0d7828064286a3cbdcc9ae51a416a5e440487ba04cf8aaace557b54709b not found: ID does not exist" containerID="3be7e0d7828064286a3cbdcc9ae51a416a5e440487ba04cf8aaace557b54709b" Oct 30 16:49:47.218705 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:47.218604 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3be7e0d7828064286a3cbdcc9ae51a416a5e440487ba04cf8aaace557b54709b"} err="failed to get container status \"3be7e0d7828064286a3cbdcc9ae51a416a5e440487ba04cf8aaace557b54709b\": rpc error: code = NotFound desc = could not find container \"3be7e0d7828064286a3cbdcc9ae51a416a5e440487ba04cf8aaace557b54709b\": container with ID starting with 3be7e0d7828064286a3cbdcc9ae51a416a5e440487ba04cf8aaace557b54709b not found: ID does not exist" Oct 30 16:49:47.218705 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:47.218627 4861 scope.go:122] "RemoveContainer" containerID="4c1e43473b8348da52eb260f72e3da580b724b460143702d7fa04250968de64d" Oct 30 16:49:47.262437 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:47.262416 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54"] Oct 30 16:49:47.274239 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:47.274215 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-569b62-69cf49d547-h2g54"] Oct 30 16:49:47.304986 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:47.304967 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf"] Oct 30 16:49:47.310670 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:47.310650 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-569b62-7f5554474d-tmrnf"] Oct 30 16:49:47.809665 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:47.809630 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824cd0d9-5af5-4c68-a4c8-62ff61852acc" path="/var/lib/kubelet/pods/824cd0d9-5af5-4c68-a4c8-62ff61852acc/volumes" Oct 30 16:49:47.810081 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:47.810064 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c35fb6ac-30a4-489b-827e-90831f6827d8" path="/var/lib/kubelet/pods/c35fb6ac-30a4-489b-827e-90831f6827d8/volumes" Oct 30 16:49:50.118750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.118691 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k"] Oct 30 16:49:50.119305 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.118962 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" podUID="f5eba3b4-3bc9-4065-8714-4ae82e8cf790" containerName="maas-api" containerID="cri-o://e83cf738c9053c395805abbb4d9f9aced4266459acfe74b930458dcb7e59c464" gracePeriod=30 Oct 30 16:49:50.291996 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:49:50.291952 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:49:50.309541 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.309292 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" podUID="a18fbfaf-05e6-40a5-8853-7465bd9e92b0" containerName="payload-processing" containerID="cri-o://e9bae21fb4bdba828b899137b97997cca063f3dd89ab814e1200c0163e1f4986" gracePeriod=25 Oct 30 16:49:50.341619 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.341587 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" podUID="b9a952e9-7858-45b7-8d4d-8cccfae8ca3c" containerName="payload-pre-processing" containerID="cri-o://370435ffe293fac8b9eb6846e55912cb319396edd43a52d79801d7f9a34b8e6a" gracePeriod=25 Oct 30 16:49:50.503811 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.503786 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:49:50.607438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.599238 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-metrics-certs\") pod \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\" (UID: \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\") " Oct 30 16:49:50.607438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.599466 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-openshift-service-ca\") pod \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\" (UID: \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\") " Oct 30 16:49:50.607438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.599499 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-maas-api-tls\") pod \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\" (UID: \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\") " Oct 30 16:49:50.607438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.599581 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-kube-api-access-dvhdj\" (UniqueName: \"kubernetes.io/projected/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-kube-api-access-dvhdj\") pod \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\" (UID: \"f5eba3b4-3bc9-4065-8714-4ae82e8cf790\") " Oct 30 16:49:50.671366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.670870 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-maas-api-tls" pod "f5eba3b4-3bc9-4065-8714-4ae82e8cf790" (UID: "f5eba3b4-3bc9-4065-8714-4ae82e8cf790"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:49:50.723531 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.710074 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:50.723531 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.720980 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-openshift-service-ca" pod "f5eba3b4-3bc9-4065-8714-4ae82e8cf790" (UID: "f5eba3b4-3bc9-4065-8714-4ae82e8cf790"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:50.742486 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.742282 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-metrics-certs" pod "f5eba3b4-3bc9-4065-8714-4ae82e8cf790" (UID: "f5eba3b4-3bc9-4065-8714-4ae82e8cf790"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:49:50.742486 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.742408 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-kube-api-access-dvhdj" pod "f5eba3b4-3bc9-4065-8714-4ae82e8cf790" (UID: "f5eba3b4-3bc9-4065-8714-4ae82e8cf790"). InnerVolumeSpecName "kube-api-access-dvhdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:50.810871 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.810842 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" Oct 30 16:49:50.811152 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.811129 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvhdj\" (UniqueName: \"kubernetes.io/projected/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-kube-api-access-dvhdj\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:50.811241 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.811158 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:50.811241 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.811168 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5eba3b4-3bc9-4065-8714-4ae82e8cf790-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:50.879094 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.879057 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" Oct 30 16:49:50.911744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.911696 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a952e9-7858-45b7-8d4d-8cccfae8ca3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a952e9-7858-45b7-8d4d-8cccfae8ca3c-plugins-config-volume\") pod \"b9a952e9-7858-45b7-8d4d-8cccfae8ca3c\" (UID: \"b9a952e9-7858-45b7-8d4d-8cccfae8ca3c\") " Oct 30 16:49:50.911934 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.911912 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a952e9-7858-45b7-8d4d-8cccfae8ca3c-kube-api-access-m8q9z\" (UniqueName: \"kubernetes.io/projected/b9a952e9-7858-45b7-8d4d-8cccfae8ca3c-kube-api-access-m8q9z\") pod \"b9a952e9-7858-45b7-8d4d-8cccfae8ca3c\" (UID: \"b9a952e9-7858-45b7-8d4d-8cccfae8ca3c\") " Oct 30 16:49:50.912369 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.912334 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a952e9-7858-45b7-8d4d-8cccfae8ca3c-plugins-config-volume" pod "b9a952e9-7858-45b7-8d4d-8cccfae8ca3c" (UID: "b9a952e9-7858-45b7-8d4d-8cccfae8ca3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:50.912543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.912524 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9a952e9-7858-45b7-8d4d-8cccfae8ca3c-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:50.914666 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:50.914641 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a952e9-7858-45b7-8d4d-8cccfae8ca3c-kube-api-access-m8q9z" pod "b9a952e9-7858-45b7-8d4d-8cccfae8ca3c" (UID: "b9a952e9-7858-45b7-8d4d-8cccfae8ca3c"). InnerVolumeSpecName "kube-api-access-m8q9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:51.013847 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.013815 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18fbfaf-05e6-40a5-8853-7465bd9e92b0-kube-api-access-5prmc\" (UniqueName: \"kubernetes.io/projected/a18fbfaf-05e6-40a5-8853-7465bd9e92b0-kube-api-access-5prmc\") pod \"a18fbfaf-05e6-40a5-8853-7465bd9e92b0\" (UID: \"a18fbfaf-05e6-40a5-8853-7465bd9e92b0\") " Oct 30 16:49:51.014045 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.013945 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18fbfaf-05e6-40a5-8853-7465bd9e92b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18fbfaf-05e6-40a5-8853-7465bd9e92b0-plugins-config-volume\") pod \"a18fbfaf-05e6-40a5-8853-7465bd9e92b0\" (UID: \"a18fbfaf-05e6-40a5-8853-7465bd9e92b0\") " Oct 30 16:49:51.014372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.014341 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18fbfaf-05e6-40a5-8853-7465bd9e92b0-plugins-config-volume" pod "a18fbfaf-05e6-40a5-8853-7465bd9e92b0" (UID: "a18fbfaf-05e6-40a5-8853-7465bd9e92b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:51.014558 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.014540 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18fbfaf-05e6-40a5-8853-7465bd9e92b0-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:51.014609 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.014564 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8q9z\" (UniqueName: \"kubernetes.io/projected/b9a952e9-7858-45b7-8d4d-8cccfae8ca3c-kube-api-access-m8q9z\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:51.016189 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.016162 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18fbfaf-05e6-40a5-8853-7465bd9e92b0-kube-api-access-5prmc" pod "a18fbfaf-05e6-40a5-8853-7465bd9e92b0" (UID: "a18fbfaf-05e6-40a5-8853-7465bd9e92b0"). InnerVolumeSpecName "kube-api-access-5prmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:51.129746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.121920 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5prmc\" (UniqueName: \"kubernetes.io/projected/a18fbfaf-05e6-40a5-8853-7465bd9e92b0-kube-api-access-5prmc\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:51.239324 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.239293 4861 generic.go:356] "Generic (PLEG): container finished" podID="a18fbfaf-05e6-40a5-8853-7465bd9e92b0" containerID="e9bae21fb4bdba828b899137b97997cca063f3dd89ab814e1200c0163e1f4986" exitCode=0 Oct 30 16:49:51.239511 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.239348 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" Oct 30 16:49:51.239511 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.239384 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" event={"ID":"a18fbfaf-05e6-40a5-8853-7465bd9e92b0","Type":"ContainerDied","Data":"e9bae21fb4bdba828b899137b97997cca063f3dd89ab814e1200c0163e1f4986"} Oct 30 16:49:51.239511 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.239439 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z" event={"ID":"a18fbfaf-05e6-40a5-8853-7465bd9e92b0","Type":"ContainerDied","Data":"f17c563dd2d2b5a464671130c35b52c25740f8f8dfc9ca9ae4f0f7dd52531de8"} Oct 30 16:49:51.239511 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.239463 4861 scope.go:122] "RemoveContainer" containerID="e9bae21fb4bdba828b899137b97997cca063f3dd89ab814e1200c0163e1f4986" Oct 30 16:49:51.241227 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.241199 4861 generic.go:356] "Generic (PLEG): container finished" podID="b9a952e9-7858-45b7-8d4d-8cccfae8ca3c" containerID="370435ffe293fac8b9eb6846e55912cb319396edd43a52d79801d7f9a34b8e6a" exitCode=0 Oct 30 16:49:51.241360 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.241233 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" event={"ID":"b9a952e9-7858-45b7-8d4d-8cccfae8ca3c","Type":"ContainerDied","Data":"370435ffe293fac8b9eb6846e55912cb319396edd43a52d79801d7f9a34b8e6a"} Oct 30 16:49:51.241360 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.241258 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" Oct 30 16:49:51.241360 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.241290 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz" event={"ID":"b9a952e9-7858-45b7-8d4d-8cccfae8ca3c","Type":"ContainerDied","Data":"d7b1f9a9fb04a9f8879b681ce17a0b5e02eb73cf09df1fc7b9c1653753f120da"} Oct 30 16:49:51.242926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.242900 4861 generic.go:356] "Generic (PLEG): container finished" podID="f5eba3b4-3bc9-4065-8714-4ae82e8cf790" containerID="e83cf738c9053c395805abbb4d9f9aced4266459acfe74b930458dcb7e59c464" exitCode=0 Oct 30 16:49:51.243026 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.242935 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" event={"ID":"f5eba3b4-3bc9-4065-8714-4ae82e8cf790","Type":"ContainerDied","Data":"e83cf738c9053c395805abbb4d9f9aced4266459acfe74b930458dcb7e59c464"} Oct 30 16:49:51.243026 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.242970 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" event={"ID":"f5eba3b4-3bc9-4065-8714-4ae82e8cf790","Type":"ContainerDied","Data":"a10981520478e2f962f53d17ff4cba5f77e00c2e8aeb8a9be0df241408c89e3f"} Oct 30 16:49:51.243120 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.243048 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k" Oct 30 16:49:51.257304 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.257281 4861 scope.go:122] "RemoveContainer" containerID="e9bae21fb4bdba828b899137b97997cca063f3dd89ab814e1200c0163e1f4986" Oct 30 16:49:51.257656 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:49:51.257635 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9bae21fb4bdba828b899137b97997cca063f3dd89ab814e1200c0163e1f4986\": container with ID starting with e9bae21fb4bdba828b899137b97997cca063f3dd89ab814e1200c0163e1f4986 not found: ID does not exist" containerID="e9bae21fb4bdba828b899137b97997cca063f3dd89ab814e1200c0163e1f4986" Oct 30 16:49:51.257778 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.257667 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9bae21fb4bdba828b899137b97997cca063f3dd89ab814e1200c0163e1f4986"} err="failed to get container status \"e9bae21fb4bdba828b899137b97997cca063f3dd89ab814e1200c0163e1f4986\": rpc error: code = NotFound desc = could not find container \"e9bae21fb4bdba828b899137b97997cca063f3dd89ab814e1200c0163e1f4986\": container with ID starting with e9bae21fb4bdba828b899137b97997cca063f3dd89ab814e1200c0163e1f4986 not found: ID does not exist" Oct 30 16:49:51.257778 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.257685 4861 scope.go:122] "RemoveContainer" containerID="370435ffe293fac8b9eb6846e55912cb319396edd43a52d79801d7f9a34b8e6a" Oct 30 16:49:51.267395 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.267368 4861 scope.go:122] "RemoveContainer" containerID="370435ffe293fac8b9eb6846e55912cb319396edd43a52d79801d7f9a34b8e6a" Oct 30 16:49:51.267701 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:49:51.267683 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370435ffe293fac8b9eb6846e55912cb319396edd43a52d79801d7f9a34b8e6a\": container with ID starting with 370435ffe293fac8b9eb6846e55912cb319396edd43a52d79801d7f9a34b8e6a not found: ID does not exist" containerID="370435ffe293fac8b9eb6846e55912cb319396edd43a52d79801d7f9a34b8e6a" Oct 30 16:49:51.267815 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.267709 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370435ffe293fac8b9eb6846e55912cb319396edd43a52d79801d7f9a34b8e6a"} err="failed to get container status \"370435ffe293fac8b9eb6846e55912cb319396edd43a52d79801d7f9a34b8e6a\": rpc error: code = NotFound desc = could not find container \"370435ffe293fac8b9eb6846e55912cb319396edd43a52d79801d7f9a34b8e6a\": container with ID starting with 370435ffe293fac8b9eb6846e55912cb319396edd43a52d79801d7f9a34b8e6a not found: ID does not exist" Oct 30 16:49:51.267815 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.267743 4861 scope.go:122] "RemoveContainer" containerID="e83cf738c9053c395805abbb4d9f9aced4266459acfe74b930458dcb7e59c464" Oct 30 16:49:51.277598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.277580 4861 scope.go:122] "RemoveContainer" containerID="e83cf738c9053c395805abbb4d9f9aced4266459acfe74b930458dcb7e59c464" Oct 30 16:49:51.277897 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:49:51.277877 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83cf738c9053c395805abbb4d9f9aced4266459acfe74b930458dcb7e59c464\": container with ID starting with e83cf738c9053c395805abbb4d9f9aced4266459acfe74b930458dcb7e59c464 not found: ID does not exist" containerID="e83cf738c9053c395805abbb4d9f9aced4266459acfe74b930458dcb7e59c464" Oct 30 16:49:51.277977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.277900 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83cf738c9053c395805abbb4d9f9aced4266459acfe74b930458dcb7e59c464"} err="failed to get container status \"e83cf738c9053c395805abbb4d9f9aced4266459acfe74b930458dcb7e59c464\": rpc error: code = NotFound desc = could not find container \"e83cf738c9053c395805abbb4d9f9aced4266459acfe74b930458dcb7e59c464\": container with ID starting with e83cf738c9053c395805abbb4d9f9aced4266459acfe74b930458dcb7e59c464 not found: ID does not exist" Oct 30 16:49:51.325442 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.325419 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz"] Oct 30 16:49:51.338118 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.338097 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-76cccb-579d949fc5-n7nvz"] Oct 30 16:49:51.372229 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.372201 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k"] Oct 30 16:49:51.376003 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.375980 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-76cccb-6c854565cd-8775k"] Oct 30 16:49:51.416634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.416602 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z"] Oct 30 16:49:51.419344 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.419325 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-76cccb-5b48547bdd-5xq7z"] Oct 30 16:49:51.808451 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.808412 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18fbfaf-05e6-40a5-8853-7465bd9e92b0" path="/var/lib/kubelet/pods/a18fbfaf-05e6-40a5-8853-7465bd9e92b0/volumes" Oct 30 16:49:51.808948 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.808926 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a952e9-7858-45b7-8d4d-8cccfae8ca3c" path="/var/lib/kubelet/pods/b9a952e9-7858-45b7-8d4d-8cccfae8ca3c/volumes" Oct 30 16:49:51.809391 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:51.809372 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5eba3b4-3bc9-4065-8714-4ae82e8cf790" path="/var/lib/kubelet/pods/f5eba3b4-3bc9-4065-8714-4ae82e8cf790/volumes" Oct 30 16:49:58.281596 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.281560 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr"] Oct 30 16:49:58.282056 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.281819 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" podUID="fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc" containerName="payload-processing" containerID="cri-o://3f89a139dcc134a3653acf0270d7792da4b32d3a5b15084a80a5869a82d121ae" gracePeriod=30 Oct 30 16:49:58.316229 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.316197 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l"] Oct 30 16:49:58.316553 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.316520 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" podUID="73abba4a-c570-436e-a00f-95fcbc80c570" containerName="payload-pre-processing" containerID="cri-o://9e67d6cef9bcf90783492172c90a45307e9baaaef2a723733b4d7606a442c7a6" gracePeriod=30 Oct 30 16:49:58.562787 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.561374 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l"] Oct 30 16:49:58.562787 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.561580 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" podUID="801f1750-fa15-4727-b81a-d7d8ff88a0d8" containerName="maas-api" containerID="cri-o://153f2944ca31b26350948ead5b68768b1b6722fe5e06ab8569f2534e53fc616a" gracePeriod=30 Oct 30 16:49:58.677083 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.677052 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" Oct 30 16:49:58.681267 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.681194 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" Oct 30 16:49:58.747082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.746108 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc-kube-api-access-b45cs\" (UniqueName: \"kubernetes.io/projected/fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc-kube-api-access-b45cs\") pod \"fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc\" (UID: \"fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc\") " Oct 30 16:49:58.747082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.746345 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73abba4a-c570-436e-a00f-95fcbc80c570-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73abba4a-c570-436e-a00f-95fcbc80c570-plugins-config-volume\") pod \"73abba4a-c570-436e-a00f-95fcbc80c570\" (UID: \"73abba4a-c570-436e-a00f-95fcbc80c570\") " Oct 30 16:49:58.747082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.746418 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73abba4a-c570-436e-a00f-95fcbc80c570-kube-api-access-jhh4j\" (UniqueName: \"kubernetes.io/projected/73abba4a-c570-436e-a00f-95fcbc80c570-kube-api-access-jhh4j\") pod \"73abba4a-c570-436e-a00f-95fcbc80c570\" (UID: \"73abba4a-c570-436e-a00f-95fcbc80c570\") " Oct 30 16:49:58.747082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.746526 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc-plugins-config-volume\") pod \"fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc\" (UID: \"fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc\") " Oct 30 16:49:58.747693 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.747562 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc-plugins-config-volume" pod "fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc" (UID: "fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:58.749412 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.749359 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73abba4a-c570-436e-a00f-95fcbc80c570-plugins-config-volume" pod "73abba4a-c570-436e-a00f-95fcbc80c570" (UID: "73abba4a-c570-436e-a00f-95fcbc80c570"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:58.752019 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.751993 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc-kube-api-access-b45cs" pod "fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc" (UID: "fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc"). InnerVolumeSpecName "kube-api-access-b45cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:58.752464 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.752439 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73abba4a-c570-436e-a00f-95fcbc80c570-kube-api-access-jhh4j" pod "73abba4a-c570-436e-a00f-95fcbc80c570" (UID: "73abba4a-c570-436e-a00f-95fcbc80c570"). InnerVolumeSpecName "kube-api-access-jhh4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:58.872996 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.872910 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b45cs\" (UniqueName: \"kubernetes.io/projected/fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc-kube-api-access-b45cs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:58.873204 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.873189 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73abba4a-c570-436e-a00f-95fcbc80c570-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:58.873306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.873292 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhh4j\" (UniqueName: \"kubernetes.io/projected/73abba4a-c570-436e-a00f-95fcbc80c570-kube-api-access-jhh4j\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:58.873413 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:58.873399 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:59.114163 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.113833 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:49:59.199543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.198105 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/801f1750-fa15-4727-b81a-d7d8ff88a0d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/801f1750-fa15-4727-b81a-d7d8ff88a0d8-metrics-certs\") pod \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\" (UID: \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\") " Oct 30 16:49:59.199543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.198225 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801f1750-fa15-4727-b81a-d7d8ff88a0d8-kube-api-access-d7sm6\" (UniqueName: \"kubernetes.io/projected/801f1750-fa15-4727-b81a-d7d8ff88a0d8-kube-api-access-d7sm6\") pod \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\" (UID: \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\") " Oct 30 16:49:59.199543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.198411 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/801f1750-fa15-4727-b81a-d7d8ff88a0d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/801f1750-fa15-4727-b81a-d7d8ff88a0d8-maas-api-tls\") pod \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\" (UID: \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\") " Oct 30 16:49:59.199543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.198521 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/801f1750-fa15-4727-b81a-d7d8ff88a0d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/801f1750-fa15-4727-b81a-d7d8ff88a0d8-openshift-service-ca\") pod \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\" (UID: \"801f1750-fa15-4727-b81a-d7d8ff88a0d8\") " Oct 30 16:49:59.202994 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.202964 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801f1750-fa15-4727-b81a-d7d8ff88a0d8-openshift-service-ca" pod "801f1750-fa15-4727-b81a-d7d8ff88a0d8" (UID: "801f1750-fa15-4727-b81a-d7d8ff88a0d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:49:59.226011 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.225967 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/801f1750-fa15-4727-b81a-d7d8ff88a0d8-maas-api-tls" pod "801f1750-fa15-4727-b81a-d7d8ff88a0d8" (UID: "801f1750-fa15-4727-b81a-d7d8ff88a0d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:49:59.229609 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.229551 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801f1750-fa15-4727-b81a-d7d8ff88a0d8-kube-api-access-d7sm6" pod "801f1750-fa15-4727-b81a-d7d8ff88a0d8" (UID: "801f1750-fa15-4727-b81a-d7d8ff88a0d8"). InnerVolumeSpecName "kube-api-access-d7sm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:49:59.230902 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.230857 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/801f1750-fa15-4727-b81a-d7d8ff88a0d8-metrics-certs" pod "801f1750-fa15-4727-b81a-d7d8ff88a0d8" (UID: "801f1750-fa15-4727-b81a-d7d8ff88a0d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:49:59.305755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.305698 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/801f1750-fa15-4727-b81a-d7d8ff88a0d8-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:59.325268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.323261 4861 generic.go:356] "Generic (PLEG): container finished" podID="fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc" containerID="3f89a139dcc134a3653acf0270d7792da4b32d3a5b15084a80a5869a82d121ae" exitCode=0 Oct 30 16:49:59.325559 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.323292 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" event={"ID":"fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc","Type":"ContainerDied","Data":"3f89a139dcc134a3653acf0270d7792da4b32d3a5b15084a80a5869a82d121ae"} Oct 30 16:49:59.329218 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.329191 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" event={"ID":"fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc","Type":"ContainerDied","Data":"3db33cf9f4818875a5b6c317409e89aeb6e82a51023bd8260841958b14e74503"} Oct 30 16:49:59.329381 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.325425 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/801f1750-fa15-4727-b81a-d7d8ff88a0d8-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:59.329510 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.329495 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7sm6\" (UniqueName: \"kubernetes.io/projected/801f1750-fa15-4727-b81a-d7d8ff88a0d8-kube-api-access-d7sm6\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:59.329630 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.329615 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/801f1750-fa15-4727-b81a-d7d8ff88a0d8-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:49:59.329750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.329686 4861 generic.go:356] "Generic (PLEG): container finished" podID="73abba4a-c570-436e-a00f-95fcbc80c570" containerID="9e67d6cef9bcf90783492172c90a45307e9baaaef2a723733b4d7606a442c7a6" exitCode=0 Oct 30 16:49:59.329925 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.329707 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" event={"ID":"73abba4a-c570-436e-a00f-95fcbc80c570","Type":"ContainerDied","Data":"9e67d6cef9bcf90783492172c90a45307e9baaaef2a723733b4d7606a442c7a6"} Oct 30 16:49:59.330006 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.329979 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" event={"ID":"73abba4a-c570-436e-a00f-95fcbc80c570","Type":"ContainerDied","Data":"d68b962f745383e05c5a84e692e6dd816a60855901439c96e2b2d38283f3248f"} Oct 30 16:49:59.330064 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.323376 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr" Oct 30 16:49:59.330064 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.329793 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l" Oct 30 16:49:59.330064 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.330046 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k"] Oct 30 16:49:59.330387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.330360 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" podUID="b5f3e0b2-750b-4545-9c66-dfc005023077" containerName="istio-proxy" containerID="cri-o://d49e68ec3dd04ebdb6f6e4a371d1e9010808c485cd46f45907055d0c95fc5687" gracePeriod=30 Oct 30 16:49:59.330896 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.329253 4861 scope.go:122] "RemoveContainer" containerID="3f89a139dcc134a3653acf0270d7792da4b32d3a5b15084a80a5869a82d121ae" Oct 30 16:49:59.343108 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.343083 4861 generic.go:356] "Generic (PLEG): container finished" podID="801f1750-fa15-4727-b81a-d7d8ff88a0d8" containerID="153f2944ca31b26350948ead5b68768b1b6722fe5e06ab8569f2534e53fc616a" exitCode=0 Oct 30 16:49:59.343251 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.343225 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" event={"ID":"801f1750-fa15-4727-b81a-d7d8ff88a0d8","Type":"ContainerDied","Data":"153f2944ca31b26350948ead5b68768b1b6722fe5e06ab8569f2534e53fc616a"} Oct 30 16:49:59.343332 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.343285 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" event={"ID":"801f1750-fa15-4727-b81a-d7d8ff88a0d8","Type":"ContainerDied","Data":"ffc497352241021e4d205203d08f27cce9911536436de52639ede4f7df49e856"} Oct 30 16:49:59.343398 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.343207 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l" Oct 30 16:49:59.352239 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.352224 4861 scope.go:122] "RemoveContainer" containerID="3f89a139dcc134a3653acf0270d7792da4b32d3a5b15084a80a5869a82d121ae" Oct 30 16:49:59.352942 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:49:59.352795 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f89a139dcc134a3653acf0270d7792da4b32d3a5b15084a80a5869a82d121ae\": container with ID starting with 3f89a139dcc134a3653acf0270d7792da4b32d3a5b15084a80a5869a82d121ae not found: ID does not exist" containerID="3f89a139dcc134a3653acf0270d7792da4b32d3a5b15084a80a5869a82d121ae" Oct 30 16:49:59.352942 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.352836 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f89a139dcc134a3653acf0270d7792da4b32d3a5b15084a80a5869a82d121ae"} err="failed to get container status \"3f89a139dcc134a3653acf0270d7792da4b32d3a5b15084a80a5869a82d121ae\": rpc error: code = NotFound desc = could not find container \"3f89a139dcc134a3653acf0270d7792da4b32d3a5b15084a80a5869a82d121ae\": container with ID starting with 3f89a139dcc134a3653acf0270d7792da4b32d3a5b15084a80a5869a82d121ae not found: ID does not exist" Oct 30 16:49:59.352942 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.352863 4861 scope.go:122] "RemoveContainer" containerID="9e67d6cef9bcf90783492172c90a45307e9baaaef2a723733b4d7606a442c7a6" Oct 30 16:49:59.364926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.364896 4861 scope.go:122] "RemoveContainer" containerID="9e67d6cef9bcf90783492172c90a45307e9baaaef2a723733b4d7606a442c7a6" Oct 30 16:49:59.365206 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:49:59.365188 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e67d6cef9bcf90783492172c90a45307e9baaaef2a723733b4d7606a442c7a6\": container with ID starting with 9e67d6cef9bcf90783492172c90a45307e9baaaef2a723733b4d7606a442c7a6 not found: ID does not exist" containerID="9e67d6cef9bcf90783492172c90a45307e9baaaef2a723733b4d7606a442c7a6" Oct 30 16:49:59.365278 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.365218 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e67d6cef9bcf90783492172c90a45307e9baaaef2a723733b4d7606a442c7a6"} err="failed to get container status \"9e67d6cef9bcf90783492172c90a45307e9baaaef2a723733b4d7606a442c7a6\": rpc error: code = NotFound desc = could not find container \"9e67d6cef9bcf90783492172c90a45307e9baaaef2a723733b4d7606a442c7a6\": container with ID starting with 9e67d6cef9bcf90783492172c90a45307e9baaaef2a723733b4d7606a442c7a6 not found: ID does not exist" Oct 30 16:49:59.365278 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.365238 4861 scope.go:122] "RemoveContainer" containerID="153f2944ca31b26350948ead5b68768b1b6722fe5e06ab8569f2534e53fc616a" Oct 30 16:49:59.374616 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.374601 4861 scope.go:122] "RemoveContainer" containerID="153f2944ca31b26350948ead5b68768b1b6722fe5e06ab8569f2534e53fc616a" Oct 30 16:49:59.374916 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:49:59.374898 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153f2944ca31b26350948ead5b68768b1b6722fe5e06ab8569f2534e53fc616a\": container with ID starting with 153f2944ca31b26350948ead5b68768b1b6722fe5e06ab8569f2534e53fc616a not found: ID does not exist" containerID="153f2944ca31b26350948ead5b68768b1b6722fe5e06ab8569f2534e53fc616a" Oct 30 16:49:59.374967 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.374923 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153f2944ca31b26350948ead5b68768b1b6722fe5e06ab8569f2534e53fc616a"} err="failed to get container status \"153f2944ca31b26350948ead5b68768b1b6722fe5e06ab8569f2534e53fc616a\": rpc error: code = NotFound desc = could not find container \"153f2944ca31b26350948ead5b68768b1b6722fe5e06ab8569f2534e53fc616a\": container with ID starting with 153f2944ca31b26350948ead5b68768b1b6722fe5e06ab8569f2534e53fc616a not found: ID does not exist" Oct 30 16:49:59.500775 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.487842 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr"] Oct 30 16:49:59.522757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.520489 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-504442-7b4cf6699b-lcxnr"] Oct 30 16:49:59.528073 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.528014 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l"] Oct 30 16:49:59.533925 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.533899 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-504442-57f65d8bff-5xt7l"] Oct 30 16:49:59.573445 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.573410 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l"] Oct 30 16:49:59.578097 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.578066 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-569b62-75cc56f8b8-cv24l"] Oct 30 16:49:59.808008 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.807936 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73abba4a-c570-436e-a00f-95fcbc80c570" path="/var/lib/kubelet/pods/73abba4a-c570-436e-a00f-95fcbc80c570/volumes" Oct 30 16:49:59.808408 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.808394 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="801f1750-fa15-4727-b81a-d7d8ff88a0d8" path="/var/lib/kubelet/pods/801f1750-fa15-4727-b81a-d7d8ff88a0d8/volumes" Oct 30 16:49:59.808828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:49:59.808813 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc" path="/var/lib/kubelet/pods/fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc/volumes" Oct 30 16:50:04.327318 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.327285 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq"] Oct 30 16:50:04.618014 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.617991 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:50:04.648185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.648155 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-workload-certs\") pod \"b5f3e0b2-750b-4545-9c66-dfc005023077\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " Oct 30 16:50:04.648320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.648197 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f3e0b2-750b-4545-9c66-dfc005023077-kube-api-access-668xc\" (UniqueName: \"kubernetes.io/projected/b5f3e0b2-750b-4545-9c66-dfc005023077-kube-api-access-668xc\") pod \"b5f3e0b2-750b-4545-9c66-dfc005023077\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " Oct 30 16:50:04.648320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.648265 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-podinfo\") pod \"b5f3e0b2-750b-4545-9c66-dfc005023077\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " Oct 30 16:50:04.648320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.648314 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-workload-socket\") pod \"b5f3e0b2-750b-4545-9c66-dfc005023077\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " Oct 30 16:50:04.648456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.648359 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-credential-socket\") pod \"b5f3e0b2-750b-4545-9c66-dfc005023077\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " Oct 30 16:50:04.648456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.648418 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-token\" (UniqueName: \"kubernetes.io/projected/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-token\") pod \"b5f3e0b2-750b-4545-9c66-dfc005023077\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " Oct 30 16:50:04.648456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.648419 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-workload-certs" pod "b5f3e0b2-750b-4545-9c66-dfc005023077" (UID: "b5f3e0b2-750b-4545-9c66-dfc005023077"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:04.648621 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.648480 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-envoy\") pod \"b5f3e0b2-750b-4545-9c66-dfc005023077\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " Oct 30 16:50:04.648621 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.648536 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5f3e0b2-750b-4545-9c66-dfc005023077-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f3e0b2-750b-4545-9c66-dfc005023077-istiod-ca-cert\") pod \"b5f3e0b2-750b-4545-9c66-dfc005023077\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " Oct 30 16:50:04.648765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.648634 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-data\") pod \"b5f3e0b2-750b-4545-9c66-dfc005023077\" (UID: \"b5f3e0b2-750b-4545-9c66-dfc005023077\") " Oct 30 16:50:04.648765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.648659 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-credential-socket" pod "b5f3e0b2-750b-4545-9c66-dfc005023077" (UID: "b5f3e0b2-750b-4545-9c66-dfc005023077"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:04.648765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.648672 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-workload-socket" pod "b5f3e0b2-750b-4545-9c66-dfc005023077" (UID: "b5f3e0b2-750b-4545-9c66-dfc005023077"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:04.648971 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.648946 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-data" pod "b5f3e0b2-750b-4545-9c66-dfc005023077" (UID: "b5f3e0b2-750b-4545-9c66-dfc005023077"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:04.649214 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.649156 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f3e0b2-750b-4545-9c66-dfc005023077-istiod-ca-cert" pod "b5f3e0b2-750b-4545-9c66-dfc005023077" (UID: "b5f3e0b2-750b-4545-9c66-dfc005023077"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:04.649303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.649280 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:04.649367 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.649310 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:04.649367 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.649327 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:04.649367 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.649342 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:04.650773 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.650716 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f3e0b2-750b-4545-9c66-dfc005023077-kube-api-access-668xc" pod "b5f3e0b2-750b-4545-9c66-dfc005023077" (UID: "b5f3e0b2-750b-4545-9c66-dfc005023077"). InnerVolumeSpecName "kube-api-access-668xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:04.650876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.650836 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-envoy" pod "b5f3e0b2-750b-4545-9c66-dfc005023077" (UID: "b5f3e0b2-750b-4545-9c66-dfc005023077"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:04.651127 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.651111 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-podinfo" pod "b5f3e0b2-750b-4545-9c66-dfc005023077" (UID: "b5f3e0b2-750b-4545-9c66-dfc005023077"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:04.651202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.651135 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-token" pod "b5f3e0b2-750b-4545-9c66-dfc005023077" (UID: "b5f3e0b2-750b-4545-9c66-dfc005023077"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:04.750642 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.750606 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-668xc\" (UniqueName: \"kubernetes.io/projected/b5f3e0b2-750b-4545-9c66-dfc005023077-kube-api-access-668xc\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:04.750642 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.750636 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:04.750642 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.750647 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:04.750918 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.750655 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5f3e0b2-750b-4545-9c66-dfc005023077-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:04.750918 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:04.750663 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5f3e0b2-750b-4545-9c66-dfc005023077-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:05.409950 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:05.409914 4861 generic.go:356] "Generic (PLEG): container finished" podID="b5f3e0b2-750b-4545-9c66-dfc005023077" containerID="d49e68ec3dd04ebdb6f6e4a371d1e9010808c485cd46f45907055d0c95fc5687" exitCode=0 Oct 30 16:50:05.410396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:05.409958 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" Oct 30 16:50:05.410396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:05.409993 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" event={"ID":"b5f3e0b2-750b-4545-9c66-dfc005023077","Type":"ContainerDied","Data":"d49e68ec3dd04ebdb6f6e4a371d1e9010808c485cd46f45907055d0c95fc5687"} Oct 30 16:50:05.410396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:05.410042 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k" event={"ID":"b5f3e0b2-750b-4545-9c66-dfc005023077","Type":"ContainerDied","Data":"97f9d1724d8b273e1762d5749ff69dd43cf7e652d13b6edcdfba7c975f317437"} Oct 30 16:50:05.410396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:05.410063 4861 scope.go:122] "RemoveContainer" containerID="d49e68ec3dd04ebdb6f6e4a371d1e9010808c485cd46f45907055d0c95fc5687" Oct 30 16:50:05.426210 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:05.426194 4861 scope.go:122] "RemoveContainer" containerID="c9659376aa2a0d6d014d9874d8e0be3857b0bbca6f16bfa835b338e7ea6e87ec" Oct 30 16:50:05.435953 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:05.435937 4861 scope.go:122] "RemoveContainer" containerID="d49e68ec3dd04ebdb6f6e4a371d1e9010808c485cd46f45907055d0c95fc5687" Oct 30 16:50:05.436248 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:50:05.436230 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d49e68ec3dd04ebdb6f6e4a371d1e9010808c485cd46f45907055d0c95fc5687\": container with ID starting with d49e68ec3dd04ebdb6f6e4a371d1e9010808c485cd46f45907055d0c95fc5687 not found: ID does not exist" containerID="d49e68ec3dd04ebdb6f6e4a371d1e9010808c485cd46f45907055d0c95fc5687" Oct 30 16:50:05.436303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:05.436257 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d49e68ec3dd04ebdb6f6e4a371d1e9010808c485cd46f45907055d0c95fc5687"} err="failed to get container status \"d49e68ec3dd04ebdb6f6e4a371d1e9010808c485cd46f45907055d0c95fc5687\": rpc error: code = NotFound desc = could not find container \"d49e68ec3dd04ebdb6f6e4a371d1e9010808c485cd46f45907055d0c95fc5687\": container with ID starting with d49e68ec3dd04ebdb6f6e4a371d1e9010808c485cd46f45907055d0c95fc5687 not found: ID does not exist" Oct 30 16:50:05.436303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:05.436276 4861 scope.go:122] "RemoveContainer" containerID="c9659376aa2a0d6d014d9874d8e0be3857b0bbca6f16bfa835b338e7ea6e87ec" Oct 30 16:50:05.436541 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:50:05.436522 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9659376aa2a0d6d014d9874d8e0be3857b0bbca6f16bfa835b338e7ea6e87ec\": container with ID starting with c9659376aa2a0d6d014d9874d8e0be3857b0bbca6f16bfa835b338e7ea6e87ec not found: ID does not exist" containerID="c9659376aa2a0d6d014d9874d8e0be3857b0bbca6f16bfa835b338e7ea6e87ec" Oct 30 16:50:05.436590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:05.436548 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9659376aa2a0d6d014d9874d8e0be3857b0bbca6f16bfa835b338e7ea6e87ec"} err="failed to get container status \"c9659376aa2a0d6d014d9874d8e0be3857b0bbca6f16bfa835b338e7ea6e87ec\": rpc error: code = NotFound desc = could not find container \"c9659376aa2a0d6d014d9874d8e0be3857b0bbca6f16bfa835b338e7ea6e87ec\": container with ID starting with c9659376aa2a0d6d014d9874d8e0be3857b0bbca6f16bfa835b338e7ea6e87ec not found: ID does not exist" Oct 30 16:50:05.477273 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:05.477237 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k"] Oct 30 16:50:05.489475 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:05.489456 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-76cccb-openshift-default-7b9cfd7cb5-8pp2k"] Oct 30 16:50:05.808411 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:05.808371 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5f3e0b2-750b-4545-9c66-dfc005023077" path="/var/lib/kubelet/pods/b5f3e0b2-750b-4545-9c66-dfc005023077/volumes" Oct 30 16:50:06.948887 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:06.948854 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7"] Oct 30 16:50:06.949364 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:06.949110 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" podUID="69254306-affa-4f98-a1a0-93c0e29c14ae" containerName="istio-proxy" containerID="cri-o://627e74539b89cc3524f26942cdbcfb5f3bda6026dc169192efd99c097c0f8743" gracePeriod=30 Oct 30 16:50:10.335618 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:10.335581 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk"] Oct 30 16:50:10.367306 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:10.367266 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd"] Oct 30 16:50:12.240155 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.240129 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:50:12.358887 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.358845 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69254306-affa-4f98-a1a0-93c0e29c14ae-kube-api-access-ltzhl\" (UniqueName: \"kubernetes.io/projected/69254306-affa-4f98-a1a0-93c0e29c14ae-kube-api-access-ltzhl\") pod \"69254306-affa-4f98-a1a0-93c0e29c14ae\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " Oct 30 16:50:12.359075 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.358913 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-data\") pod \"69254306-affa-4f98-a1a0-93c0e29c14ae\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " Oct 30 16:50:12.359075 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.359025 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-workload-certs\") pod \"69254306-affa-4f98-a1a0-93c0e29c14ae\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " Oct 30 16:50:12.359202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.359090 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-podinfo\") pod \"69254306-affa-4f98-a1a0-93c0e29c14ae\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " Oct 30 16:50:12.359202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.359121 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-envoy\") pod \"69254306-affa-4f98-a1a0-93c0e29c14ae\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " Oct 30 16:50:12.359202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.359166 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69254306-affa-4f98-a1a0-93c0e29c14ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69254306-affa-4f98-a1a0-93c0e29c14ae-istiod-ca-cert\") pod \"69254306-affa-4f98-a1a0-93c0e29c14ae\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " Oct 30 16:50:12.359202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.359195 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-token\" (UniqueName: \"kubernetes.io/projected/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-token\") pod \"69254306-affa-4f98-a1a0-93c0e29c14ae\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " Oct 30 16:50:12.359432 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.359259 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-credential-socket\") pod \"69254306-affa-4f98-a1a0-93c0e29c14ae\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " Oct 30 16:50:12.359432 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.359311 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-workload-socket\") pod \"69254306-affa-4f98-a1a0-93c0e29c14ae\" (UID: \"69254306-affa-4f98-a1a0-93c0e29c14ae\") " Oct 30 16:50:12.359432 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.359312 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-data" pod "69254306-affa-4f98-a1a0-93c0e29c14ae" (UID: "69254306-affa-4f98-a1a0-93c0e29c14ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:12.359827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.359799 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69254306-affa-4f98-a1a0-93c0e29c14ae-istiod-ca-cert" pod "69254306-affa-4f98-a1a0-93c0e29c14ae" (UID: "69254306-affa-4f98-a1a0-93c0e29c14ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:12.360088 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.360015 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-credential-socket" pod "69254306-affa-4f98-a1a0-93c0e29c14ae" (UID: "69254306-affa-4f98-a1a0-93c0e29c14ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:12.360088 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.360036 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-workload-socket" pod "69254306-affa-4f98-a1a0-93c0e29c14ae" (UID: "69254306-affa-4f98-a1a0-93c0e29c14ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:12.360233 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.360129 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:12.360233 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.360147 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69254306-affa-4f98-a1a0-93c0e29c14ae-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:12.360348 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.360246 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-workload-certs" pod "69254306-affa-4f98-a1a0-93c0e29c14ae" (UID: "69254306-affa-4f98-a1a0-93c0e29c14ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:12.361763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.361709 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-token" pod "69254306-affa-4f98-a1a0-93c0e29c14ae" (UID: "69254306-affa-4f98-a1a0-93c0e29c14ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:12.362012 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.361983 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69254306-affa-4f98-a1a0-93c0e29c14ae-kube-api-access-ltzhl" pod "69254306-affa-4f98-a1a0-93c0e29c14ae" (UID: "69254306-affa-4f98-a1a0-93c0e29c14ae"). InnerVolumeSpecName "kube-api-access-ltzhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:12.362116 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.362098 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-podinfo" pod "69254306-affa-4f98-a1a0-93c0e29c14ae" (UID: "69254306-affa-4f98-a1a0-93c0e29c14ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:12.362230 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.362210 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-envoy" pod "69254306-affa-4f98-a1a0-93c0e29c14ae" (UID: "69254306-affa-4f98-a1a0-93c0e29c14ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:12.460444 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.460407 4861 generic.go:356] "Generic (PLEG): container finished" podID="69254306-affa-4f98-a1a0-93c0e29c14ae" containerID="627e74539b89cc3524f26942cdbcfb5f3bda6026dc169192efd99c097c0f8743" exitCode=0 Oct 30 16:50:12.460608 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.460471 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" Oct 30 16:50:12.460608 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.460504 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" event={"ID":"69254306-affa-4f98-a1a0-93c0e29c14ae","Type":"ContainerDied","Data":"627e74539b89cc3524f26942cdbcfb5f3bda6026dc169192efd99c097c0f8743"} Oct 30 16:50:12.460608 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.460576 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7" event={"ID":"69254306-affa-4f98-a1a0-93c0e29c14ae","Type":"ContainerDied","Data":"f806052ad1f31206d4ba151c11573d4aa9135e2bdb17ec4e506a7ffc35384f60"} Oct 30 16:50:12.460717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.460612 4861 scope.go:122] "RemoveContainer" containerID="627e74539b89cc3524f26942cdbcfb5f3bda6026dc169192efd99c097c0f8743" Oct 30 16:50:12.461191 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.461172 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltzhl\" (UniqueName: \"kubernetes.io/projected/69254306-affa-4f98-a1a0-93c0e29c14ae-kube-api-access-ltzhl\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:12.461191 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.461192 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:12.461354 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.461201 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:12.461354 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.461210 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:12.461354 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.461222 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69254306-affa-4f98-a1a0-93c0e29c14ae-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:12.461354 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.461230 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:12.461354 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.461240 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69254306-affa-4f98-a1a0-93c0e29c14ae-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:12.473095 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.473080 4861 scope.go:122] "RemoveContainer" containerID="95c397d46b637a08e031934f0b569949f844b2983e3de282b6aa6eb51db87c3e" Oct 30 16:50:12.482963 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.482944 4861 scope.go:122] "RemoveContainer" containerID="627e74539b89cc3524f26942cdbcfb5f3bda6026dc169192efd99c097c0f8743" Oct 30 16:50:12.483249 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:50:12.483227 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"627e74539b89cc3524f26942cdbcfb5f3bda6026dc169192efd99c097c0f8743\": container with ID starting with 627e74539b89cc3524f26942cdbcfb5f3bda6026dc169192efd99c097c0f8743 not found: ID does not exist" containerID="627e74539b89cc3524f26942cdbcfb5f3bda6026dc169192efd99c097c0f8743" Oct 30 16:50:12.483330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.483256 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627e74539b89cc3524f26942cdbcfb5f3bda6026dc169192efd99c097c0f8743"} err="failed to get container status \"627e74539b89cc3524f26942cdbcfb5f3bda6026dc169192efd99c097c0f8743\": rpc error: code = NotFound desc = could not find container \"627e74539b89cc3524f26942cdbcfb5f3bda6026dc169192efd99c097c0f8743\": container with ID starting with 627e74539b89cc3524f26942cdbcfb5f3bda6026dc169192efd99c097c0f8743 not found: ID does not exist" Oct 30 16:50:12.483330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.483272 4861 scope.go:122] "RemoveContainer" containerID="95c397d46b637a08e031934f0b569949f844b2983e3de282b6aa6eb51db87c3e" Oct 30 16:50:12.483592 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:50:12.483568 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c397d46b637a08e031934f0b569949f844b2983e3de282b6aa6eb51db87c3e\": container with ID starting with 95c397d46b637a08e031934f0b569949f844b2983e3de282b6aa6eb51db87c3e not found: ID does not exist" containerID="95c397d46b637a08e031934f0b569949f844b2983e3de282b6aa6eb51db87c3e" Oct 30 16:50:12.483633 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.483587 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c397d46b637a08e031934f0b569949f844b2983e3de282b6aa6eb51db87c3e"} err="failed to get container status \"95c397d46b637a08e031934f0b569949f844b2983e3de282b6aa6eb51db87c3e\": rpc error: code = NotFound desc = could not find container \"95c397d46b637a08e031934f0b569949f844b2983e3de282b6aa6eb51db87c3e\": container with ID starting with 95c397d46b637a08e031934f0b569949f844b2983e3de282b6aa6eb51db87c3e not found: ID does not exist" Oct 30 16:50:12.515520 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.515458 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7"] Oct 30 16:50:12.519929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:12.519905 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-569b62-openshift-default-7847bfb9df-m9pp7"] Oct 30 16:50:13.808083 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:13.808048 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69254306-affa-4f98-a1a0-93c0e29c14ae" path="/var/lib/kubelet/pods/69254306-affa-4f98-a1a0-93c0e29c14ae/volumes" Oct 30 16:50:15.336699 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.336657 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" podUID="35ece6d8-cd04-4890-a3e2-a865b649d884" containerName="payload-processing" containerID="cri-o://80a735db863d915efaa362fa0ade3fca3d15c30d54a50ebee065c320e1b16936" gracePeriod=25 Oct 30 16:50:15.368240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.368211 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" podUID="b66fbeb1-84ba-4fbf-b25c-21256006e1c9" containerName="payload-pre-processing" containerID="cri-o://48e52a80601e41ec6a609f211eeba9d7ef31d15a1dd30c2a27ff87e2d987026f" gracePeriod=25 Oct 30 16:50:15.530818 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.527446 4861 generic.go:356] "Generic (PLEG): container finished" podID="35ece6d8-cd04-4890-a3e2-a865b649d884" containerID="80a735db863d915efaa362fa0ade3fca3d15c30d54a50ebee065c320e1b16936" exitCode=0 Oct 30 16:50:15.530818 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.527571 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" event={"ID":"35ece6d8-cd04-4890-a3e2-a865b649d884","Type":"ContainerDied","Data":"80a735db863d915efaa362fa0ade3fca3d15c30d54a50ebee065c320e1b16936"} Oct 30 16:50:15.530818 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.529601 4861 generic.go:356] "Generic (PLEG): container finished" podID="b66fbeb1-84ba-4fbf-b25c-21256006e1c9" containerID="48e52a80601e41ec6a609f211eeba9d7ef31d15a1dd30c2a27ff87e2d987026f" exitCode=0 Oct 30 16:50:15.530818 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.529663 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" event={"ID":"b66fbeb1-84ba-4fbf-b25c-21256006e1c9","Type":"ContainerDied","Data":"48e52a80601e41ec6a609f211eeba9d7ef31d15a1dd30c2a27ff87e2d987026f"} Oct 30 16:50:15.668461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.668440 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" Oct 30 16:50:15.704949 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.704925 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" Oct 30 16:50:15.817650 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.817623 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35ece6d8-cd04-4890-a3e2-a865b649d884-kube-api-access-g2vrt\" (UniqueName: \"kubernetes.io/projected/35ece6d8-cd04-4890-a3e2-a865b649d884-kube-api-access-g2vrt\") pod \"35ece6d8-cd04-4890-a3e2-a865b649d884\" (UID: \"35ece6d8-cd04-4890-a3e2-a865b649d884\") " Oct 30 16:50:15.817848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.817686 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35ece6d8-cd04-4890-a3e2-a865b649d884-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ece6d8-cd04-4890-a3e2-a865b649d884-plugins-config-volume\") pod \"35ece6d8-cd04-4890-a3e2-a865b649d884\" (UID: \"35ece6d8-cd04-4890-a3e2-a865b649d884\") " Oct 30 16:50:15.817848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.817714 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66fbeb1-84ba-4fbf-b25c-21256006e1c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66fbeb1-84ba-4fbf-b25c-21256006e1c9-plugins-config-volume\") pod \"b66fbeb1-84ba-4fbf-b25c-21256006e1c9\" (UID: \"b66fbeb1-84ba-4fbf-b25c-21256006e1c9\") " Oct 30 16:50:15.817947 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.817914 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66fbeb1-84ba-4fbf-b25c-21256006e1c9-kube-api-access-mq826\" (UniqueName: \"kubernetes.io/projected/b66fbeb1-84ba-4fbf-b25c-21256006e1c9-kube-api-access-mq826\") pod \"b66fbeb1-84ba-4fbf-b25c-21256006e1c9\" (UID: \"b66fbeb1-84ba-4fbf-b25c-21256006e1c9\") " Oct 30 16:50:15.818149 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.818120 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35ece6d8-cd04-4890-a3e2-a865b649d884-plugins-config-volume" pod "35ece6d8-cd04-4890-a3e2-a865b649d884" (UID: "35ece6d8-cd04-4890-a3e2-a865b649d884"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:15.818271 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.818150 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66fbeb1-84ba-4fbf-b25c-21256006e1c9-plugins-config-volume" pod "b66fbeb1-84ba-4fbf-b25c-21256006e1c9" (UID: "b66fbeb1-84ba-4fbf-b25c-21256006e1c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:15.818386 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.818371 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35ece6d8-cd04-4890-a3e2-a865b649d884-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:15.818425 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.818389 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66fbeb1-84ba-4fbf-b25c-21256006e1c9-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:15.820001 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.819975 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ece6d8-cd04-4890-a3e2-a865b649d884-kube-api-access-g2vrt" pod "35ece6d8-cd04-4890-a3e2-a865b649d884" (UID: "35ece6d8-cd04-4890-a3e2-a865b649d884"). InnerVolumeSpecName "kube-api-access-g2vrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:15.820100 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.820043 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66fbeb1-84ba-4fbf-b25c-21256006e1c9-kube-api-access-mq826" pod "b66fbeb1-84ba-4fbf-b25c-21256006e1c9" (UID: "b66fbeb1-84ba-4fbf-b25c-21256006e1c9"). InnerVolumeSpecName "kube-api-access-mq826". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:15.919362 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.919298 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq826\" (UniqueName: \"kubernetes.io/projected/b66fbeb1-84ba-4fbf-b25c-21256006e1c9-kube-api-access-mq826\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:15.919362 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:15.919323 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2vrt\" (UniqueName: \"kubernetes.io/projected/35ece6d8-cd04-4890-a3e2-a865b649d884-kube-api-access-g2vrt\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:16.538052 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:16.538021 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" Oct 30 16:50:16.538052 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:16.538044 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk" event={"ID":"35ece6d8-cd04-4890-a3e2-a865b649d884","Type":"ContainerDied","Data":"43100f160f8c62de7ede7af7b19cac170a73c11daae8b81352183ede373e8597"} Oct 30 16:50:16.538566 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:16.538110 4861 scope.go:122] "RemoveContainer" containerID="80a735db863d915efaa362fa0ade3fca3d15c30d54a50ebee065c320e1b16936" Oct 30 16:50:16.539677 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:16.539648 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" event={"ID":"b66fbeb1-84ba-4fbf-b25c-21256006e1c9","Type":"ContainerDied","Data":"db0919b2177163fd2373599ba723383b0140d8afb5ab80be6a1e8f92d5e652a6"} Oct 30 16:50:16.539832 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:16.539681 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd" Oct 30 16:50:16.553420 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:16.553395 4861 scope.go:122] "RemoveContainer" containerID="48e52a80601e41ec6a609f211eeba9d7ef31d15a1dd30c2a27ff87e2d987026f" Oct 30 16:50:16.593880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:16.593853 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk"] Oct 30 16:50:16.606207 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:16.606137 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ea8dca-56f48b6b6-25rjk"] Oct 30 16:50:16.611761 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:16.611734 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq"] Oct 30 16:50:16.612023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:16.611996 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" podUID="a22008ff-d02b-47ba-bc61-eb8302e0cd3d" containerName="istio-proxy" containerID="cri-o://2d582dc6d79eed83186c8c59e14fd1f1587f9dd53d8d35895b17c98b98c07ccc" gracePeriod=30 Oct 30 16:50:16.659233 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:16.659198 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd"] Oct 30 16:50:16.664308 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:16.664280 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ea8dca-6df7c46559-72drd"] Oct 30 16:50:17.034165 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:17.034129 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74"] Oct 30 16:50:17.074673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:17.074644 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms"] Oct 30 16:50:17.810551 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:17.810516 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35ece6d8-cd04-4890-a3e2-a865b649d884" path="/var/lib/kubelet/pods/35ece6d8-cd04-4890-a3e2-a865b649d884/volumes" Oct 30 16:50:17.811185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:17.811163 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66fbeb1-84ba-4fbf-b25c-21256006e1c9" path="/var/lib/kubelet/pods/b66fbeb1-84ba-4fbf-b25c-21256006e1c9/volumes" Oct 30 16:50:19.349936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.349901 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" podUID="579c56b7-c963-4ced-b809-1e40ff5fcbf5" containerName="main" containerID="cri-o://0efeb7fc47bc828bfe02027318c1d109e5d24bae68e9ad706b461b20c2276f93" gracePeriod=45 Oct 30 16:50:19.565578 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.565546 4861 generic.go:356] "Generic (PLEG): container finished" podID="579c56b7-c963-4ced-b809-1e40ff5fcbf5" containerID="0efeb7fc47bc828bfe02027318c1d109e5d24bae68e9ad706b461b20c2276f93" exitCode=0 Oct 30 16:50:19.565744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.565631 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" event={"ID":"579c56b7-c963-4ced-b809-1e40ff5fcbf5","Type":"ContainerDied","Data":"0efeb7fc47bc828bfe02027318c1d109e5d24bae68e9ad706b461b20c2276f93"} Oct 30 16:50:19.633503 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.633480 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:50:19.790815 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.790779 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-home\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-home\") pod \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " Oct 30 16:50:19.790995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.790836 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-dshm\") pod \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " Oct 30 16:50:19.790995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.790939 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/579c56b7-c963-4ced-b809-1e40ff5fcbf5-kube-api-access-4xf5h\" (UniqueName: \"kubernetes.io/projected/579c56b7-c963-4ced-b809-1e40ff5fcbf5-kube-api-access-4xf5h\") pod \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " Oct 30 16:50:19.790995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.790968 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-model-cache\") pod \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " Oct 30 16:50:19.791187 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.791022 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/579c56b7-c963-4ced-b809-1e40ff5fcbf5-tls-certs\" (UniqueName: \"kubernetes.io/secret/579c56b7-c963-4ced-b809-1e40ff5fcbf5-tls-certs\") pod \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " Oct 30 16:50:19.791187 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.791133 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-tmp-dir\") pod \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\" (UID: \"579c56b7-c963-4ced-b809-1e40ff5fcbf5\") " Oct 30 16:50:19.791491 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.791421 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-home" pod "579c56b7-c963-4ced-b809-1e40ff5fcbf5" (UID: "579c56b7-c963-4ced-b809-1e40ff5fcbf5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:19.791997 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.791790 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-tmp-dir" pod "579c56b7-c963-4ced-b809-1e40ff5fcbf5" (UID: "579c56b7-c963-4ced-b809-1e40ff5fcbf5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:19.791997 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.791952 4861 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:19.791997 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.791971 4861 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:19.792261 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.792071 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-model-cache" pod "579c56b7-c963-4ced-b809-1e40ff5fcbf5" (UID: "579c56b7-c963-4ced-b809-1e40ff5fcbf5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:19.793779 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.793705 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-dshm" pod "579c56b7-c963-4ced-b809-1e40ff5fcbf5" (UID: "579c56b7-c963-4ced-b809-1e40ff5fcbf5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:19.793930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.793912 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/579c56b7-c963-4ced-b809-1e40ff5fcbf5-tls-certs" pod "579c56b7-c963-4ced-b809-1e40ff5fcbf5" (UID: "579c56b7-c963-4ced-b809-1e40ff5fcbf5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:19.794162 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.794145 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/579c56b7-c963-4ced-b809-1e40ff5fcbf5-kube-api-access-4xf5h" pod "579c56b7-c963-4ced-b809-1e40ff5fcbf5" (UID: "579c56b7-c963-4ced-b809-1e40ff5fcbf5"). InnerVolumeSpecName "kube-api-access-4xf5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:19.893382 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.893328 4861 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:19.893382 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.893353 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xf5h\" (UniqueName: \"kubernetes.io/projected/579c56b7-c963-4ced-b809-1e40ff5fcbf5-kube-api-access-4xf5h\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:19.893382 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.893363 4861 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/579c56b7-c963-4ced-b809-1e40ff5fcbf5-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:19.893382 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:19.893372 4861 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/579c56b7-c963-4ced-b809-1e40ff5fcbf5-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:20.291452 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:50:20.291417 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:50:20.480462 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:20.480430 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr"] Oct 30 16:50:20.511755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:20.511703 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b"] Oct 30 16:50:20.578004 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:20.577909 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" event={"ID":"579c56b7-c963-4ced-b809-1e40ff5fcbf5","Type":"ContainerDied","Data":"477da58ebee192e5b755c1480149d0d1840e12fa298302a3d804339c38c9da56"} Oct 30 16:50:20.578004 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:20.577962 4861 scope.go:122] "RemoveContainer" containerID="0efeb7fc47bc828bfe02027318c1d109e5d24bae68e9ad706b461b20c2276f93" Oct 30 16:50:20.578246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:20.578084 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq" Oct 30 16:50:20.657531 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:20.657496 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq"] Oct 30 16:50:20.663594 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:20.663565 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-504442/ipp-route-504442-kserve-69995f5fc7-bkjvq"] Oct 30 16:50:21.811428 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.811396 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="579c56b7-c963-4ced-b809-1e40ff5fcbf5" path="/var/lib/kubelet/pods/579c56b7-c963-4ced-b809-1e40ff5fcbf5/volumes" Oct 30 16:50:21.876441 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.876416 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:50:21.918503 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.918441 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-token\" (UniqueName: \"kubernetes.io/projected/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-token\") pod \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " Oct 30 16:50:21.918628 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.918532 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-credential-socket\") pod \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " Oct 30 16:50:21.918628 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.918553 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-workload-certs\") pod \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " Oct 30 16:50:21.918628 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.918576 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-kube-api-access-j28mj\" (UniqueName: \"kubernetes.io/projected/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-kube-api-access-j28mj\") pod \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " Oct 30 16:50:21.918628 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.918604 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istiod-ca-cert\") pod \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " Oct 30 16:50:21.918858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.918635 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-podinfo\") pod \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " Oct 30 16:50:21.918858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.918759 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-envoy\") pod \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " Oct 30 16:50:21.918858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.918843 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-data\") pod \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " Oct 30 16:50:21.919024 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.918889 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-workload-socket\") pod \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\" (UID: \"a22008ff-d02b-47ba-bc61-eb8302e0cd3d\") " Oct 30 16:50:21.919024 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.918898 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-credential-socket" pod "a22008ff-d02b-47ba-bc61-eb8302e0cd3d" (UID: "a22008ff-d02b-47ba-bc61-eb8302e0cd3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:21.919024 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.918922 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-workload-certs" pod "a22008ff-d02b-47ba-bc61-eb8302e0cd3d" (UID: "a22008ff-d02b-47ba-bc61-eb8302e0cd3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:21.919184 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.919159 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-workload-socket" pod "a22008ff-d02b-47ba-bc61-eb8302e0cd3d" (UID: "a22008ff-d02b-47ba-bc61-eb8302e0cd3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:21.919184 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.919172 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-data" pod "a22008ff-d02b-47ba-bc61-eb8302e0cd3d" (UID: "a22008ff-d02b-47ba-bc61-eb8302e0cd3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:21.919285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.919196 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istiod-ca-cert" pod "a22008ff-d02b-47ba-bc61-eb8302e0cd3d" (UID: "a22008ff-d02b-47ba-bc61-eb8302e0cd3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:21.919439 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.919412 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:21.919439 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.919435 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:21.919627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.919448 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:21.919627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.919461 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:21.919627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.919472 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:21.920983 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.920962 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-token" pod "a22008ff-d02b-47ba-bc61-eb8302e0cd3d" (UID: "a22008ff-d02b-47ba-bc61-eb8302e0cd3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:21.921110 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.921094 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-envoy" pod "a22008ff-d02b-47ba-bc61-eb8302e0cd3d" (UID: "a22008ff-d02b-47ba-bc61-eb8302e0cd3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:21.921584 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.921561 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-podinfo" pod "a22008ff-d02b-47ba-bc61-eb8302e0cd3d" (UID: "a22008ff-d02b-47ba-bc61-eb8302e0cd3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:21.921660 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:21.921593 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-kube-api-access-j28mj" pod "a22008ff-d02b-47ba-bc61-eb8302e0cd3d" (UID: "a22008ff-d02b-47ba-bc61-eb8302e0cd3d"). InnerVolumeSpecName "kube-api-access-j28mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:22.021245 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.021215 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:22.021245 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.021242 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:22.021245 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.021251 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j28mj\" (UniqueName: \"kubernetes.io/projected/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-kube-api-access-j28mj\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:22.021514 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.021259 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22008ff-d02b-47ba-bc61-eb8302e0cd3d-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:22.035311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.035269 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" podUID="d5a2c326-4e32-4826-b24d-b9ea22ef5e3e" containerName="payload-processing" containerID="cri-o://64b4f519412477e1d2a91995c7f643b872f67124320431c5a92f64a4c1c89341" gracePeriod=25 Oct 30 16:50:22.076005 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.075968 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" podUID="0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5" containerName="payload-pre-processing" containerID="cri-o://440a5e8ff59121692c4d6d87ad1cb41eb7c9c2c5c24b93803328df68fcda36a5" gracePeriod=25 Oct 30 16:50:22.372904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.372871 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" Oct 30 16:50:22.414664 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.414621 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" Oct 30 16:50:22.430991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.430961 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5-plugins-config-volume\") pod \"0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5\" (UID: \"0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5\") " Oct 30 16:50:22.431164 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.431117 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a2c326-4e32-4826-b24d-b9ea22ef5e3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a2c326-4e32-4826-b24d-b9ea22ef5e3e-plugins-config-volume\") pod \"d5a2c326-4e32-4826-b24d-b9ea22ef5e3e\" (UID: \"d5a2c326-4e32-4826-b24d-b9ea22ef5e3e\") " Oct 30 16:50:22.431242 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.431223 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a2c326-4e32-4826-b24d-b9ea22ef5e3e-kube-api-access-6th8h\" (UniqueName: \"kubernetes.io/projected/d5a2c326-4e32-4826-b24d-b9ea22ef5e3e-kube-api-access-6th8h\") pod \"d5a2c326-4e32-4826-b24d-b9ea22ef5e3e\" (UID: \"d5a2c326-4e32-4826-b24d-b9ea22ef5e3e\") " Oct 30 16:50:22.432158 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.431307 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5-kube-api-access-6hdw2\" (UniqueName: \"kubernetes.io/projected/0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5-kube-api-access-6hdw2\") pod \"0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5\" (UID: \"0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5\") " Oct 30 16:50:22.432158 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.431346 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5-plugins-config-volume" pod "0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5" (UID: "0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:22.432158 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.431633 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a2c326-4e32-4826-b24d-b9ea22ef5e3e-plugins-config-volume" pod "d5a2c326-4e32-4826-b24d-b9ea22ef5e3e" (UID: "d5a2c326-4e32-4826-b24d-b9ea22ef5e3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:22.432158 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.432119 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a2c326-4e32-4826-b24d-b9ea22ef5e3e-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:22.432158 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.432137 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:22.434431 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.434410 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a2c326-4e32-4826-b24d-b9ea22ef5e3e-kube-api-access-6th8h" pod "d5a2c326-4e32-4826-b24d-b9ea22ef5e3e" (UID: "d5a2c326-4e32-4826-b24d-b9ea22ef5e3e"). InnerVolumeSpecName "kube-api-access-6th8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:22.434521 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.434429 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5-kube-api-access-6hdw2" pod "0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5" (UID: "0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5"). InnerVolumeSpecName "kube-api-access-6hdw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:22.533370 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.533334 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6th8h\" (UniqueName: \"kubernetes.io/projected/d5a2c326-4e32-4826-b24d-b9ea22ef5e3e-kube-api-access-6th8h\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:22.533370 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.533369 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hdw2\" (UniqueName: \"kubernetes.io/projected/0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5-kube-api-access-6hdw2\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:22.594544 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.594509 4861 generic.go:356] "Generic (PLEG): container finished" podID="0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5" containerID="440a5e8ff59121692c4d6d87ad1cb41eb7c9c2c5c24b93803328df68fcda36a5" exitCode=0 Oct 30 16:50:22.594681 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.594597 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" Oct 30 16:50:22.594681 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.594613 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" event={"ID":"0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5","Type":"ContainerDied","Data":"440a5e8ff59121692c4d6d87ad1cb41eb7c9c2c5c24b93803328df68fcda36a5"} Oct 30 16:50:22.594681 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.594666 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms" event={"ID":"0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5","Type":"ContainerDied","Data":"4a426ef953ae72196e55e69f09dd3bde313f9d881e2981466c8ea22c557fe505"} Oct 30 16:50:22.594876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.594689 4861 scope.go:122] "RemoveContainer" containerID="440a5e8ff59121692c4d6d87ad1cb41eb7c9c2c5c24b93803328df68fcda36a5" Oct 30 16:50:22.596252 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.596229 4861 generic.go:356] "Generic (PLEG): container finished" podID="a22008ff-d02b-47ba-bc61-eb8302e0cd3d" containerID="2d582dc6d79eed83186c8c59e14fd1f1587f9dd53d8d35895b17c98b98c07ccc" exitCode=0 Oct 30 16:50:22.596368 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.596350 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" Oct 30 16:50:22.596438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.596422 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" event={"ID":"a22008ff-d02b-47ba-bc61-eb8302e0cd3d","Type":"ContainerDied","Data":"2d582dc6d79eed83186c8c59e14fd1f1587f9dd53d8d35895b17c98b98c07ccc"} Oct 30 16:50:22.596481 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.596467 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq" event={"ID":"a22008ff-d02b-47ba-bc61-eb8302e0cd3d","Type":"ContainerDied","Data":"26b6e6a5baf31b4b8bddd710cec25511fefa91bfd8895c61963f3db5efcf7237"} Oct 30 16:50:22.598235 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.598209 4861 generic.go:356] "Generic (PLEG): container finished" podID="d5a2c326-4e32-4826-b24d-b9ea22ef5e3e" containerID="64b4f519412477e1d2a91995c7f643b872f67124320431c5a92f64a4c1c89341" exitCode=0 Oct 30 16:50:22.598354 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.598248 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" Oct 30 16:50:22.598354 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.598292 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" event={"ID":"d5a2c326-4e32-4826-b24d-b9ea22ef5e3e","Type":"ContainerDied","Data":"64b4f519412477e1d2a91995c7f643b872f67124320431c5a92f64a4c1c89341"} Oct 30 16:50:22.598354 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.598345 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74" event={"ID":"d5a2c326-4e32-4826-b24d-b9ea22ef5e3e","Type":"ContainerDied","Data":"f97fda372654ddc1adbdcc311b87f711978d23b88c4e5f9970450cd8f601fde4"} Oct 30 16:50:22.607634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.607616 4861 scope.go:122] "RemoveContainer" containerID="440a5e8ff59121692c4d6d87ad1cb41eb7c9c2c5c24b93803328df68fcda36a5" Oct 30 16:50:22.608090 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:50:22.607999 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"440a5e8ff59121692c4d6d87ad1cb41eb7c9c2c5c24b93803328df68fcda36a5\": container with ID starting with 440a5e8ff59121692c4d6d87ad1cb41eb7c9c2c5c24b93803328df68fcda36a5 not found: ID does not exist" containerID="440a5e8ff59121692c4d6d87ad1cb41eb7c9c2c5c24b93803328df68fcda36a5" Oct 30 16:50:22.608090 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.608033 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"440a5e8ff59121692c4d6d87ad1cb41eb7c9c2c5c24b93803328df68fcda36a5"} err="failed to get container status \"440a5e8ff59121692c4d6d87ad1cb41eb7c9c2c5c24b93803328df68fcda36a5\": rpc error: code = NotFound desc = could not find container \"440a5e8ff59121692c4d6d87ad1cb41eb7c9c2c5c24b93803328df68fcda36a5\": container with ID starting with 440a5e8ff59121692c4d6d87ad1cb41eb7c9c2c5c24b93803328df68fcda36a5 not found: ID does not exist" Oct 30 16:50:22.608090 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.608059 4861 scope.go:122] "RemoveContainer" containerID="2d582dc6d79eed83186c8c59e14fd1f1587f9dd53d8d35895b17c98b98c07ccc" Oct 30 16:50:22.620134 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.620116 4861 scope.go:122] "RemoveContainer" containerID="2d582dc6d79eed83186c8c59e14fd1f1587f9dd53d8d35895b17c98b98c07ccc" Oct 30 16:50:22.620390 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:50:22.620368 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d582dc6d79eed83186c8c59e14fd1f1587f9dd53d8d35895b17c98b98c07ccc\": container with ID starting with 2d582dc6d79eed83186c8c59e14fd1f1587f9dd53d8d35895b17c98b98c07ccc not found: ID does not exist" containerID="2d582dc6d79eed83186c8c59e14fd1f1587f9dd53d8d35895b17c98b98c07ccc" Oct 30 16:50:22.620478 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.620399 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d582dc6d79eed83186c8c59e14fd1f1587f9dd53d8d35895b17c98b98c07ccc"} err="failed to get container status \"2d582dc6d79eed83186c8c59e14fd1f1587f9dd53d8d35895b17c98b98c07ccc\": rpc error: code = NotFound desc = could not find container \"2d582dc6d79eed83186c8c59e14fd1f1587f9dd53d8d35895b17c98b98c07ccc\": container with ID starting with 2d582dc6d79eed83186c8c59e14fd1f1587f9dd53d8d35895b17c98b98c07ccc not found: ID does not exist" Oct 30 16:50:22.620478 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.620422 4861 scope.go:122] "RemoveContainer" containerID="64b4f519412477e1d2a91995c7f643b872f67124320431c5a92f64a4c1c89341" Oct 30 16:50:22.629547 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.629526 4861 scope.go:122] "RemoveContainer" containerID="64b4f519412477e1d2a91995c7f643b872f67124320431c5a92f64a4c1c89341" Oct 30 16:50:22.629887 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:50:22.629869 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b4f519412477e1d2a91995c7f643b872f67124320431c5a92f64a4c1c89341\": container with ID starting with 64b4f519412477e1d2a91995c7f643b872f67124320431c5a92f64a4c1c89341 not found: ID does not exist" containerID="64b4f519412477e1d2a91995c7f643b872f67124320431c5a92f64a4c1c89341" Oct 30 16:50:22.629958 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.629892 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b4f519412477e1d2a91995c7f643b872f67124320431c5a92f64a4c1c89341"} err="failed to get container status \"64b4f519412477e1d2a91995c7f643b872f67124320431c5a92f64a4c1c89341\": rpc error: code = NotFound desc = could not find container \"64b4f519412477e1d2a91995c7f643b872f67124320431c5a92f64a4c1c89341\": container with ID starting with 64b4f519412477e1d2a91995c7f643b872f67124320431c5a92f64a4c1c89341 not found: ID does not exist" Oct 30 16:50:22.652089 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.652068 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms"] Oct 30 16:50:22.654952 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.654933 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-c7bdd8cd7-2c2ms"] Oct 30 16:50:22.719821 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.719795 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74"] Oct 30 16:50:22.723230 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.723211 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d5903-6485468d64-44j74"] Oct 30 16:50:22.757950 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.757930 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq"] Oct 30 16:50:22.762310 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:22.762290 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-294lq"] Oct 30 16:50:23.817275 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:23.817243 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5" path="/var/lib/kubelet/pods/0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5/volumes" Oct 30 16:50:23.817708 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:23.817693 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22008ff-d02b-47ba-bc61-eb8302e0cd3d" path="/var/lib/kubelet/pods/a22008ff-d02b-47ba-bc61-eb8302e0cd3d/volumes" Oct 30 16:50:23.818077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:23.818064 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a2c326-4e32-4826-b24d-b9ea22ef5e3e" path="/var/lib/kubelet/pods/d5a2c326-4e32-4826-b24d-b9ea22ef5e3e/volumes" Oct 30 16:50:25.480800 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.480708 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" podUID="ad3ed68e-c3d2-4c25-a388-3c00d78128c1" containerName="payload-processing" containerID="cri-o://9f150c0ceb4f027cfea93f7c53ffc55bf3be6370e2d8fa1d1230a82eebac01f2" gracePeriod=25 Oct 30 16:50:25.512483 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.512446 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" podUID="fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848" containerName="payload-pre-processing" containerID="cri-o://eb48ea0da9d0c1050ddb896c5fbdd1f646fc81a20d7390145f6ee6fcf47d43cc" gracePeriod=25 Oct 30 16:50:25.636700 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.636672 4861 generic.go:356] "Generic (PLEG): container finished" podID="ad3ed68e-c3d2-4c25-a388-3c00d78128c1" containerID="9f150c0ceb4f027cfea93f7c53ffc55bf3be6370e2d8fa1d1230a82eebac01f2" exitCode=0 Oct 30 16:50:25.636836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.636767 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" event={"ID":"ad3ed68e-c3d2-4c25-a388-3c00d78128c1","Type":"ContainerDied","Data":"9f150c0ceb4f027cfea93f7c53ffc55bf3be6370e2d8fa1d1230a82eebac01f2"} Oct 30 16:50:25.638760 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.638735 4861 generic.go:356] "Generic (PLEG): container finished" podID="fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848" containerID="eb48ea0da9d0c1050ddb896c5fbdd1f646fc81a20d7390145f6ee6fcf47d43cc" exitCode=0 Oct 30 16:50:25.638865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.638776 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" event={"ID":"fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848","Type":"ContainerDied","Data":"eb48ea0da9d0c1050ddb896c5fbdd1f646fc81a20d7390145f6ee6fcf47d43cc"} Oct 30 16:50:25.806492 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.806465 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" Oct 30 16:50:25.840423 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.840397 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" Oct 30 16:50:25.911082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.910284 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848-kube-api-access-s8ctj\" (UniqueName: \"kubernetes.io/projected/fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848-kube-api-access-s8ctj\") pod \"fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848\" (UID: \"fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848\") " Oct 30 16:50:25.911082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.910413 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848-plugins-config-volume\") pod \"fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848\" (UID: \"fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848\") " Oct 30 16:50:25.911082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.910479 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3ed68e-c3d2-4c25-a388-3c00d78128c1-kube-api-access-hsbtc\" (UniqueName: \"kubernetes.io/projected/ad3ed68e-c3d2-4c25-a388-3c00d78128c1-kube-api-access-hsbtc\") pod \"ad3ed68e-c3d2-4c25-a388-3c00d78128c1\" (UID: \"ad3ed68e-c3d2-4c25-a388-3c00d78128c1\") " Oct 30 16:50:25.911082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.910600 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3ed68e-c3d2-4c25-a388-3c00d78128c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3ed68e-c3d2-4c25-a388-3c00d78128c1-plugins-config-volume\") pod \"ad3ed68e-c3d2-4c25-a388-3c00d78128c1\" (UID: \"ad3ed68e-c3d2-4c25-a388-3c00d78128c1\") " Oct 30 16:50:25.911082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.911069 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848-plugins-config-volume" pod "fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848" (UID: "fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:25.911494 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.911120 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3ed68e-c3d2-4c25-a388-3c00d78128c1-plugins-config-volume" pod "ad3ed68e-c3d2-4c25-a388-3c00d78128c1" (UID: "ad3ed68e-c3d2-4c25-a388-3c00d78128c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:25.911494 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.911244 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:25.911494 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.911266 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad3ed68e-c3d2-4c25-a388-3c00d78128c1-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:25.913315 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.913288 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3ed68e-c3d2-4c25-a388-3c00d78128c1-kube-api-access-hsbtc" pod "ad3ed68e-c3d2-4c25-a388-3c00d78128c1" (UID: "ad3ed68e-c3d2-4c25-a388-3c00d78128c1"). InnerVolumeSpecName "kube-api-access-hsbtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:25.913551 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:25.913528 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848-kube-api-access-s8ctj" pod "fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848" (UID: "fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848"). InnerVolumeSpecName "kube-api-access-s8ctj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:26.012637 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:26.012561 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsbtc\" (UniqueName: \"kubernetes.io/projected/ad3ed68e-c3d2-4c25-a388-3c00d78128c1-kube-api-access-hsbtc\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.012637 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:26.012589 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8ctj\" (UniqueName: \"kubernetes.io/projected/fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848-kube-api-access-s8ctj\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.646826 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:26.646795 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" Oct 30 16:50:26.647243 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:26.646822 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b" event={"ID":"fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848","Type":"ContainerDied","Data":"0a25c05490bd677d8212c9024aa9130ed25de2b4d3a76051aea484a48dc2cb76"} Oct 30 16:50:26.647243 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:26.646883 4861 scope.go:122] "RemoveContainer" containerID="eb48ea0da9d0c1050ddb896c5fbdd1f646fc81a20d7390145f6ee6fcf47d43cc" Oct 30 16:50:26.648474 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:26.648442 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" event={"ID":"ad3ed68e-c3d2-4c25-a388-3c00d78128c1","Type":"ContainerDied","Data":"34f6bdade904f53f87d30727d667e40cc240265cbac7cb1179ee1e06a6af2482"} Oct 30 16:50:26.648602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:26.648451 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr" Oct 30 16:50:26.657362 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:26.657346 4861 scope.go:122] "RemoveContainer" containerID="9f150c0ceb4f027cfea93f7c53ffc55bf3be6370e2d8fa1d1230a82eebac01f2" Oct 30 16:50:26.701580 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:26.701555 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr"] Oct 30 16:50:26.713999 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:26.713973 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-72b379-7cfd998596-97vhr"] Oct 30 16:50:26.741428 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:26.741406 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b"] Oct 30 16:50:26.747515 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:26.747492 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-72b379-6d4c6cb9c5-kfk7b"] Oct 30 16:50:27.807448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:27.807417 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3ed68e-c3d2-4c25-a388-3c00d78128c1" path="/var/lib/kubelet/pods/ad3ed68e-c3d2-4c25-a388-3c00d78128c1/volumes" Oct 30 16:50:27.807841 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:27.807828 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848" path="/var/lib/kubelet/pods/fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848/volumes" Oct 30 16:50:38.247519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.247412 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr"] Oct 30 16:50:38.256419 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.256386 4861 cpu_manager.go:414] "containerMap: removing container" podUID="69254306-affa-4f98-a1a0-93c0e29c14ae" containerName="istio-proxy" Oct 30 16:50:38.256652 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.256632 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69254306-affa-4f98-a1a0-93c0e29c14ae" containerName="istio-proxy" Oct 30 16:50:38.256795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.256781 4861 cpu_manager.go:414] "containerMap: removing container" podUID="a18fbfaf-05e6-40a5-8853-7465bd9e92b0" containerName="payload-processing" Oct 30 16:50:38.256921 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.256878 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18fbfaf-05e6-40a5-8853-7465bd9e92b0" containerName="payload-processing" Oct 30 16:50:38.257028 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.257017 4861 cpu_manager.go:414] "containerMap: removing container" podUID="ad3ed68e-c3d2-4c25-a388-3c00d78128c1" containerName="payload-processing" Oct 30 16:50:38.257116 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.257104 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3ed68e-c3d2-4c25-a388-3c00d78128c1" containerName="payload-processing" Oct 30 16:50:38.257235 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.257203 4861 cpu_manager.go:414] "containerMap: removing container" podUID="d5a2c326-4e32-4826-b24d-b9ea22ef5e3e" containerName="payload-processing" Oct 30 16:50:38.257330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.257319 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a2c326-4e32-4826-b24d-b9ea22ef5e3e" containerName="payload-processing" Oct 30 16:50:38.257411 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.257403 4861 cpu_manager.go:414] "containerMap: removing container" podUID="fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc" containerName="payload-processing" Oct 30 16:50:38.257472 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.257464 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc" containerName="payload-processing" Oct 30 16:50:38.257558 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.257538 4861 cpu_manager.go:414] "containerMap: removing container" podUID="fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848" containerName="payload-pre-processing" Oct 30 16:50:38.257645 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.257632 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848" containerName="payload-pre-processing" Oct 30 16:50:38.257748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.257739 4861 cpu_manager.go:414] "containerMap: removing container" podUID="12ea7c70-5553-4086-b4f5-dc5895c4f0a1" containerName="istio-proxy" Oct 30 16:50:38.257834 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.257823 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ea7c70-5553-4086-b4f5-dc5895c4f0a1" containerName="istio-proxy" Oct 30 16:50:38.257914 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.257904 4861 cpu_manager.go:414] "containerMap: removing container" podUID="35ece6d8-cd04-4890-a3e2-a865b649d884" containerName="payload-processing" Oct 30 16:50:38.258025 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.258011 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35ece6d8-cd04-4890-a3e2-a865b649d884" containerName="payload-processing" Oct 30 16:50:38.258139 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.258129 4861 cpu_manager.go:414] "containerMap: removing container" podUID="c35fb6ac-30a4-489b-827e-90831f6827d8" containerName="payload-processing" Oct 30 16:50:38.258214 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.258204 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c35fb6ac-30a4-489b-827e-90831f6827d8" containerName="payload-processing" Oct 30 16:50:38.258289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.258280 4861 cpu_manager.go:414] "containerMap: removing container" podUID="579c56b7-c963-4ced-b809-1e40ff5fcbf5" containerName="main" Oct 30 16:50:38.266816 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.266790 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="579c56b7-c963-4ced-b809-1e40ff5fcbf5" containerName="main" Oct 30 16:50:38.266984 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.266972 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b9a952e9-7858-45b7-8d4d-8cccfae8ca3c" containerName="payload-pre-processing" Oct 30 16:50:38.267086 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.267071 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a952e9-7858-45b7-8d4d-8cccfae8ca3c" containerName="payload-pre-processing" Oct 30 16:50:38.267175 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.267165 4861 cpu_manager.go:414] "containerMap: removing container" podUID="12ea7c70-5553-4086-b4f5-dc5895c4f0a1" containerName="istio-proxy" Oct 30 16:50:38.267246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.267238 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12ea7c70-5553-4086-b4f5-dc5895c4f0a1" containerName="istio-proxy" Oct 30 16:50:38.267319 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.267310 4861 cpu_manager.go:414] "containerMap: removing container" podUID="824cd0d9-5af5-4c68-a4c8-62ff61852acc" containerName="payload-pre-processing" Oct 30 16:50:38.267379 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.267371 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824cd0d9-5af5-4c68-a4c8-62ff61852acc" containerName="payload-pre-processing" Oct 30 16:50:38.267456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.267448 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b5f3e0b2-750b-4545-9c66-dfc005023077" containerName="istio-proxy" Oct 30 16:50:38.267515 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.267507 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f3e0b2-750b-4545-9c66-dfc005023077" containerName="istio-proxy" Oct 30 16:50:38.267582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.267575 4861 cpu_manager.go:414] "containerMap: removing container" podUID="801f1750-fa15-4727-b81a-d7d8ff88a0d8" containerName="maas-api" Oct 30 16:50:38.267644 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.267637 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="801f1750-fa15-4727-b81a-d7d8ff88a0d8" containerName="maas-api" Oct 30 16:50:38.267711 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.267703 4861 cpu_manager.go:414] "containerMap: removing container" podUID="73abba4a-c570-436e-a00f-95fcbc80c570" containerName="payload-pre-processing" Oct 30 16:50:38.267801 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.267789 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73abba4a-c570-436e-a00f-95fcbc80c570" containerName="payload-pre-processing" Oct 30 16:50:38.267873 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.267866 4861 cpu_manager.go:414] "containerMap: removing container" podUID="f5eba3b4-3bc9-4065-8714-4ae82e8cf790" containerName="maas-api" Oct 30 16:50:38.267933 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.267925 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5eba3b4-3bc9-4065-8714-4ae82e8cf790" containerName="maas-api" Oct 30 16:50:38.267997 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.267989 4861 cpu_manager.go:414] "containerMap: removing container" podUID="a22008ff-d02b-47ba-bc61-eb8302e0cd3d" containerName="istio-proxy" Oct 30 16:50:38.268061 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.268053 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22008ff-d02b-47ba-bc61-eb8302e0cd3d" containerName="istio-proxy" Oct 30 16:50:38.268123 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.268115 4861 cpu_manager.go:414] "containerMap: removing container" podUID="0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5" containerName="payload-pre-processing" Oct 30 16:50:38.268182 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.268174 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5" containerName="payload-pre-processing" Oct 30 16:50:38.268847 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.268247 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b66fbeb1-84ba-4fbf-b25c-21256006e1c9" containerName="payload-pre-processing" Oct 30 16:50:38.269004 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.268988 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66fbeb1-84ba-4fbf-b25c-21256006e1c9" containerName="payload-pre-processing" Oct 30 16:50:38.269086 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.269077 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b5f3e0b2-750b-4545-9c66-dfc005023077" containerName="istio-proxy" Oct 30 16:50:38.269160 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.269150 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f3e0b2-750b-4545-9c66-dfc005023077" containerName="istio-proxy" Oct 30 16:50:38.269424 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.269408 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a2c326-4e32-4826-b24d-b9ea22ef5e3e" containerName="payload-processing" Oct 30 16:50:38.269537 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.269527 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef8f9bf-96a1-430f-b3d8-6e9cab2f93dc" containerName="payload-processing" Oct 30 16:50:38.269640 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.269627 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd9a9bee-f0b9-4951-aeaa-40b6f2c9d848" containerName="payload-pre-processing" Oct 30 16:50:38.269764 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.269753 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ea7c70-5553-4086-b4f5-dc5895c4f0a1" containerName="istio-proxy" Oct 30 16:50:38.269854 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.269844 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="35ece6d8-cd04-4890-a3e2-a865b649d884" containerName="payload-processing" Oct 30 16:50:38.269942 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.269932 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="c35fb6ac-30a4-489b-827e-90831f6827d8" containerName="payload-processing" Oct 30 16:50:38.270023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.270012 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="579c56b7-c963-4ced-b809-1e40ff5fcbf5" containerName="main" Oct 30 16:50:38.270104 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.270094 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a952e9-7858-45b7-8d4d-8cccfae8ca3c" containerName="payload-pre-processing" Oct 30 16:50:38.270203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.270192 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="12ea7c70-5553-4086-b4f5-dc5895c4f0a1" containerName="istio-proxy" Oct 30 16:50:38.270299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.270289 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="824cd0d9-5af5-4c68-a4c8-62ff61852acc" containerName="payload-pre-processing" Oct 30 16:50:38.270439 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.270414 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f3e0b2-750b-4545-9c66-dfc005023077" containerName="istio-proxy" Oct 30 16:50:38.270502 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.270471 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="69254306-affa-4f98-a1a0-93c0e29c14ae" containerName="istio-proxy" Oct 30 16:50:38.270502 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.270485 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="801f1750-fa15-4727-b81a-d7d8ff88a0d8" containerName="maas-api" Oct 30 16:50:38.270502 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.270496 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="73abba4a-c570-436e-a00f-95fcbc80c570" containerName="payload-pre-processing" Oct 30 16:50:38.270632 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.270509 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5eba3b4-3bc9-4065-8714-4ae82e8cf790" containerName="maas-api" Oct 30 16:50:38.270632 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.270523 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22008ff-d02b-47ba-bc61-eb8302e0cd3d" containerName="istio-proxy" Oct 30 16:50:38.270632 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.270546 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab0021f-772f-4ea2-ae0a-ab59aac7c5f5" containerName="payload-pre-processing" Oct 30 16:50:38.270632 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.270557 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66fbeb1-84ba-4fbf-b25c-21256006e1c9" containerName="payload-pre-processing" Oct 30 16:50:38.270632 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.270570 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18fbfaf-05e6-40a5-8853-7465bd9e92b0" containerName="payload-processing" Oct 30 16:50:38.270632 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.270580 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3ed68e-c3d2-4c25-a388-3c00d78128c1" containerName="payload-processing" Oct 30 16:50:38.293049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.279927 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" Oct 30 16:50:38.295578 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.295540 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr"] Oct 30 16:50:38.296315 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.296290 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6d5903\"" Oct 30 16:50:38.312782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.312637 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6d5903-dockercfg-s92jf\"" Oct 30 16:50:38.373395 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.373352 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n"] Oct 30 16:50:38.373974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.373960 4861 cpu_manager.go:414] "containerMap: removing container" podUID="69254306-affa-4f98-a1a0-93c0e29c14ae" containerName="istio-proxy" Oct 30 16:50:38.374036 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.373975 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69254306-affa-4f98-a1a0-93c0e29c14ae" containerName="istio-proxy" Oct 30 16:50:38.374073 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.374058 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f3e0b2-750b-4545-9c66-dfc005023077" containerName="istio-proxy" Oct 30 16:50:38.374073 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.374067 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="69254306-affa-4f98-a1a0-93c0e29c14ae" containerName="istio-proxy" Oct 30 16:50:38.377464 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.377444 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" Oct 30 16:50:38.398901 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.398872 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n"] Oct 30 16:50:38.409661 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.409633 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61c6199-58ad-4ff1-b662-0fa4f2d9576a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr\" (UID: \"d61c6199-58ad-4ff1-b662-0fa4f2d9576a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" Oct 30 16:50:38.409822 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.409676 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9bvr\" (UniqueName: \"kubernetes.io/projected/d61c6199-58ad-4ff1-b662-0fa4f2d9576a-kube-api-access-z9bvr\") pod \"payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr\" (UID: \"d61c6199-58ad-4ff1-b662-0fa4f2d9576a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" Oct 30 16:50:38.511132 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.511067 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sh4w\" (UniqueName: \"kubernetes.io/projected/e80e615e-5678-4264-a1ac-7bd053f88f3f-kube-api-access-5sh4w\") pod \"payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n\" (UID: \"e80e615e-5678-4264-a1ac-7bd053f88f3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" Oct 30 16:50:38.511259 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.511216 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61c6199-58ad-4ff1-b662-0fa4f2d9576a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr\" (UID: \"d61c6199-58ad-4ff1-b662-0fa4f2d9576a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" Oct 30 16:50:38.511259 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.511252 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9bvr\" (UniqueName: \"kubernetes.io/projected/d61c6199-58ad-4ff1-b662-0fa4f2d9576a-kube-api-access-z9bvr\") pod \"payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr\" (UID: \"d61c6199-58ad-4ff1-b662-0fa4f2d9576a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" Oct 30 16:50:38.511338 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.511298 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80e615e-5678-4264-a1ac-7bd053f88f3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n\" (UID: \"e80e615e-5678-4264-a1ac-7bd053f88f3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" Oct 30 16:50:38.511869 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.511852 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61c6199-58ad-4ff1-b662-0fa4f2d9576a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr\" (UID: \"d61c6199-58ad-4ff1-b662-0fa4f2d9576a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" Oct 30 16:50:38.535641 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.535606 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9bvr\" (UniqueName: \"kubernetes.io/projected/d61c6199-58ad-4ff1-b662-0fa4f2d9576a-kube-api-access-z9bvr\") pod \"payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr\" (UID: \"d61c6199-58ad-4ff1-b662-0fa4f2d9576a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" Oct 30 16:50:38.612535 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.612507 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80e615e-5678-4264-a1ac-7bd053f88f3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n\" (UID: \"e80e615e-5678-4264-a1ac-7bd053f88f3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" Oct 30 16:50:38.612682 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.612561 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sh4w\" (UniqueName: \"kubernetes.io/projected/e80e615e-5678-4264-a1ac-7bd053f88f3f-kube-api-access-5sh4w\") pod \"payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n\" (UID: \"e80e615e-5678-4264-a1ac-7bd053f88f3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" Oct 30 16:50:38.613356 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.613334 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80e615e-5678-4264-a1ac-7bd053f88f3f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n\" (UID: \"e80e615e-5678-4264-a1ac-7bd053f88f3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" Oct 30 16:50:38.620358 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.620336 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" Oct 30 16:50:38.637029 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.637000 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sh4w\" (UniqueName: \"kubernetes.io/projected/e80e615e-5678-4264-a1ac-7bd053f88f3f-kube-api-access-5sh4w\") pod \"payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n\" (UID: \"e80e615e-5678-4264-a1ac-7bd053f88f3f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" Oct 30 16:50:38.702028 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.701992 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" Oct 30 16:50:38.928697 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:38.928669 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr"] Oct 30 16:50:38.929819 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:50:38.929795 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61c6199_58ad_4ff1_b662_0fa4f2d9576a.slice/crio-8bc050d743c9a9b555044d0e7a6802a3a29ae5ef52f731c3f5d64f4ac062b75e WatchSource:0}: Error finding container 8bc050d743c9a9b555044d0e7a6802a3a29ae5ef52f731c3f5d64f4ac062b75e: Status 404 returned error can't find the container with id 8bc050d743c9a9b555044d0e7a6802a3a29ae5ef52f731c3f5d64f4ac062b75e Oct 30 16:50:39.026863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:39.026832 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n"] Oct 30 16:50:39.026986 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:50:39.026883 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80e615e_5678_4264_a1ac_7bd053f88f3f.slice/crio-9e5f8816c790d92961696dea407316ae27821a1b0daac3220d6715de68a82972 WatchSource:0}: Error finding container 9e5f8816c790d92961696dea407316ae27821a1b0daac3220d6715de68a82972: Status 404 returned error can't find the container with id 9e5f8816c790d92961696dea407316ae27821a1b0daac3220d6715de68a82972 Oct 30 16:50:39.813437 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:39.813406 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" event={"ID":"d61c6199-58ad-4ff1-b662-0fa4f2d9576a","Type":"ContainerStarted","Data":"670e456513bf4c0a59da12777eeca52e1de4ac6f7b5fe32818c0691200104472"} Oct 30 16:50:39.813982 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:39.813450 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" event={"ID":"d61c6199-58ad-4ff1-b662-0fa4f2d9576a","Type":"ContainerStarted","Data":"8bc050d743c9a9b555044d0e7a6802a3a29ae5ef52f731c3f5d64f4ac062b75e"} Oct 30 16:50:39.813982 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:39.813535 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" Oct 30 16:50:39.815264 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:39.815243 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" event={"ID":"e80e615e-5678-4264-a1ac-7bd053f88f3f","Type":"ContainerStarted","Data":"e4aff3d5879fe7d87ae2b0924cfa8df063adef9f402f9dacd8861acd0c3cbb00"} Oct 30 16:50:39.815365 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:39.815276 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" event={"ID":"e80e615e-5678-4264-a1ac-7bd053f88f3f","Type":"ContainerStarted","Data":"9e5f8816c790d92961696dea407316ae27821a1b0daac3220d6715de68a82972"} Oct 30 16:50:39.815365 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:39.815295 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" Oct 30 16:50:39.856274 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:39.856211 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" podStartSLOduration=1.85619558 podStartE2EDuration="1.85619558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:39.855215219 +0000 UTC m=+1670.197889258" watchObservedRunningTime="2026-10-30 16:50:39.85619558 +0000 UTC m=+1670.198869620" Oct 30 16:50:39.899873 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:39.899832 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" podStartSLOduration=1.899819528 podStartE2EDuration="1.899819528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:39.897234727 +0000 UTC m=+1670.239908766" watchObservedRunningTime="2026-10-30 16:50:39.899819528 +0000 UTC m=+1670.242493601" Oct 30 16:50:43.050104 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.050067 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b"] Oct 30 16:50:43.050618 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.050323 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" podUID="331f3eb5-061d-40f1-9ff2-9f1e90e3619c" containerName="maas-api" containerID="cri-o://52ba391142cb0fcf6b7c39fccdd05eda23f17c0d59fe1130dcac13a92b1c4ec4" gracePeriod=30 Oct 30 16:50:43.398953 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.398916 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:50:43.584085 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.583976 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-metrics-certs\") pod \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\" (UID: \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\") " Oct 30 16:50:43.584085 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.584065 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-openshift-service-ca\") pod \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\" (UID: \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\") " Oct 30 16:50:43.584344 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.584123 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-kube-api-access-xjg6z\" (UniqueName: \"kubernetes.io/projected/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-kube-api-access-xjg6z\") pod \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\" (UID: \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\") " Oct 30 16:50:43.584344 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.584223 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-maas-api-tls\") pod \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\" (UID: \"331f3eb5-061d-40f1-9ff2-9f1e90e3619c\") " Oct 30 16:50:43.587452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.586802 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-openshift-service-ca" pod "331f3eb5-061d-40f1-9ff2-9f1e90e3619c" (UID: "331f3eb5-061d-40f1-9ff2-9f1e90e3619c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:43.593850 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.592887 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-maas-api-tls" pod "331f3eb5-061d-40f1-9ff2-9f1e90e3619c" (UID: "331f3eb5-061d-40f1-9ff2-9f1e90e3619c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:43.593850 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.593032 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-kube-api-access-xjg6z" pod "331f3eb5-061d-40f1-9ff2-9f1e90e3619c" (UID: "331f3eb5-061d-40f1-9ff2-9f1e90e3619c"). InnerVolumeSpecName "kube-api-access-xjg6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:43.612899 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.604566 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-metrics-certs" pod "331f3eb5-061d-40f1-9ff2-9f1e90e3619c" (UID: "331f3eb5-061d-40f1-9ff2-9f1e90e3619c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:43.690161 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.690040 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:43.690161 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.690085 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:43.690161 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.690102 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjg6z\" (UniqueName: \"kubernetes.io/projected/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-kube-api-access-xjg6z\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:43.690161 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.690118 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/331f3eb5-061d-40f1-9ff2-9f1e90e3619c-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:43.847875 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.847784 4861 generic.go:356] "Generic (PLEG): container finished" podID="331f3eb5-061d-40f1-9ff2-9f1e90e3619c" containerID="52ba391142cb0fcf6b7c39fccdd05eda23f17c0d59fe1130dcac13a92b1c4ec4" exitCode=0 Oct 30 16:50:43.847875 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.847846 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" Oct 30 16:50:43.847875 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.847874 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" event={"ID":"331f3eb5-061d-40f1-9ff2-9f1e90e3619c","Type":"ContainerDied","Data":"52ba391142cb0fcf6b7c39fccdd05eda23f17c0d59fe1130dcac13a92b1c4ec4"} Oct 30 16:50:43.848150 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.847950 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b" event={"ID":"331f3eb5-061d-40f1-9ff2-9f1e90e3619c","Type":"ContainerDied","Data":"510cfc537aeb231f7d2fca9c0dcedd832c4b64f52c9e932b8364bff9fc733c96"} Oct 30 16:50:43.848150 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.848003 4861 scope.go:122] "RemoveContainer" containerID="52ba391142cb0fcf6b7c39fccdd05eda23f17c0d59fe1130dcac13a92b1c4ec4" Oct 30 16:50:43.861574 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.861552 4861 scope.go:122] "RemoveContainer" containerID="52ba391142cb0fcf6b7c39fccdd05eda23f17c0d59fe1130dcac13a92b1c4ec4" Oct 30 16:50:43.861932 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:50:43.861908 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52ba391142cb0fcf6b7c39fccdd05eda23f17c0d59fe1130dcac13a92b1c4ec4\": container with ID starting with 52ba391142cb0fcf6b7c39fccdd05eda23f17c0d59fe1130dcac13a92b1c4ec4 not found: ID does not exist" containerID="52ba391142cb0fcf6b7c39fccdd05eda23f17c0d59fe1130dcac13a92b1c4ec4" Oct 30 16:50:43.862002 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.861940 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ba391142cb0fcf6b7c39fccdd05eda23f17c0d59fe1130dcac13a92b1c4ec4"} err="failed to get container status \"52ba391142cb0fcf6b7c39fccdd05eda23f17c0d59fe1130dcac13a92b1c4ec4\": rpc error: code = NotFound desc = could not find container \"52ba391142cb0fcf6b7c39fccdd05eda23f17c0d59fe1130dcac13a92b1c4ec4\": container with ID starting with 52ba391142cb0fcf6b7c39fccdd05eda23f17c0d59fe1130dcac13a92b1c4ec4 not found: ID does not exist" Oct 30 16:50:43.902119 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.902081 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b"] Oct 30 16:50:43.913926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:43.913895 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-72b379-5f6b56655d-lhk5b"] Oct 30 16:50:45.808079 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:45.808042 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="331f3eb5-061d-40f1-9ff2-9f1e90e3619c" path="/var/lib/kubelet/pods/331f3eb5-061d-40f1-9ff2-9f1e90e3619c/volumes" Oct 30 16:50:49.316347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:49.312625 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5"] Oct 30 16:50:49.316347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:49.312910 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" podUID="ab025762-9af1-4c25-a0cc-4ab9c0221987" containerName="istio-proxy" containerID="cri-o://145ae9daaea2719442e60284123893434d1252b076cf2620bd82f9b919872c02" gracePeriod=30 Oct 30 16:50:50.047717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:50.047660 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc"] Oct 30 16:50:50.048066 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:50.048019 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" podUID="e3c4a663-826a-4d68-9208-e16cb76839b5" containerName="istio-proxy" containerID="cri-o://8f9ca42bd98cf870601928f2955f9e5320a291767a6df1bd19505c5379cedb8f" gracePeriod=30 Oct 30 16:50:50.293449 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:50:50.293404 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:50:50.825172 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:50.825143 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" Oct 30 16:50:50.825585 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:50.825231 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" Oct 30 16:50:51.074856 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:51.074778 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:50:51.075014 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:51.074855 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" podUID="ab025762-9af1-4c25-a0cc-4ab9c0221987" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:50:52.124804 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:52.124774 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj"] Oct 30 16:50:52.125380 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:52.125345 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" podUID="9db4fb72-7b88-4237-8c83-38f14f803caa" containerName="istio-proxy" containerID="cri-o://9ee60eb06b52bc87e0ff7ff965610e4d6f5696bd10a3b8eda36f23cae36f6ff2" gracePeriod=30 Oct 30 16:50:54.639229 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.639203 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:50:54.742240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.742204 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-data\") pod \"ab025762-9af1-4c25-a0cc-4ab9c0221987\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " Oct 30 16:50:54.742405 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.742282 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-envoy\") pod \"ab025762-9af1-4c25-a0cc-4ab9c0221987\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " Oct 30 16:50:54.742405 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.742346 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-workload-certs\") pod \"ab025762-9af1-4c25-a0cc-4ab9c0221987\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " Oct 30 16:50:54.742523 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.742419 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-token\" (UniqueName: \"kubernetes.io/projected/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-token\") pod \"ab025762-9af1-4c25-a0cc-4ab9c0221987\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " Oct 30 16:50:54.742523 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.742480 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab025762-9af1-4c25-a0cc-4ab9c0221987-kube-api-access-rh4rw\" (UniqueName: \"kubernetes.io/projected/ab025762-9af1-4c25-a0cc-4ab9c0221987-kube-api-access-rh4rw\") pod \"ab025762-9af1-4c25-a0cc-4ab9c0221987\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " Oct 30 16:50:54.742523 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.742510 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-workload-socket\") pod \"ab025762-9af1-4c25-a0cc-4ab9c0221987\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " Oct 30 16:50:54.742692 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.742527 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-data" pod "ab025762-9af1-4c25-a0cc-4ab9c0221987" (UID: "ab025762-9af1-4c25-a0cc-4ab9c0221987"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:54.742692 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.742572 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab025762-9af1-4c25-a0cc-4ab9c0221987-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab025762-9af1-4c25-a0cc-4ab9c0221987-istiod-ca-cert\") pod \"ab025762-9af1-4c25-a0cc-4ab9c0221987\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " Oct 30 16:50:54.742692 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.742623 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-podinfo\") pod \"ab025762-9af1-4c25-a0cc-4ab9c0221987\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " Oct 30 16:50:54.742692 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.742669 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-credential-socket\") pod \"ab025762-9af1-4c25-a0cc-4ab9c0221987\" (UID: \"ab025762-9af1-4c25-a0cc-4ab9c0221987\") " Oct 30 16:50:54.743084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.743061 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-workload-socket" pod "ab025762-9af1-4c25-a0cc-4ab9c0221987" (UID: "ab025762-9af1-4c25-a0cc-4ab9c0221987"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:54.743166 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.743090 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab025762-9af1-4c25-a0cc-4ab9c0221987-istiod-ca-cert" pod "ab025762-9af1-4c25-a0cc-4ab9c0221987" (UID: "ab025762-9af1-4c25-a0cc-4ab9c0221987"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:54.743313 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.743291 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-credential-socket" pod "ab025762-9af1-4c25-a0cc-4ab9c0221987" (UID: "ab025762-9af1-4c25-a0cc-4ab9c0221987"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:54.743387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.743368 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:54.743485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.743394 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab025762-9af1-4c25-a0cc-4ab9c0221987-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:54.743485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.743409 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:54.743660 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.743619 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-workload-certs" pod "ab025762-9af1-4c25-a0cc-4ab9c0221987" (UID: "ab025762-9af1-4c25-a0cc-4ab9c0221987"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:54.745582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.745538 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab025762-9af1-4c25-a0cc-4ab9c0221987-kube-api-access-rh4rw" pod "ab025762-9af1-4c25-a0cc-4ab9c0221987" (UID: "ab025762-9af1-4c25-a0cc-4ab9c0221987"). InnerVolumeSpecName "kube-api-access-rh4rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:54.745840 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.745819 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-envoy" pod "ab025762-9af1-4c25-a0cc-4ab9c0221987" (UID: "ab025762-9af1-4c25-a0cc-4ab9c0221987"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:54.745956 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.745934 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-podinfo" pod "ab025762-9af1-4c25-a0cc-4ab9c0221987" (UID: "ab025762-9af1-4c25-a0cc-4ab9c0221987"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:54.746150 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.746126 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-token" pod "ab025762-9af1-4c25-a0cc-4ab9c0221987" (UID: "ab025762-9af1-4c25-a0cc-4ab9c0221987"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:54.844415 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.844386 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:54.844415 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.844414 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:54.844632 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.844427 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh4rw\" (UniqueName: \"kubernetes.io/projected/ab025762-9af1-4c25-a0cc-4ab9c0221987-kube-api-access-rh4rw\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:54.844632 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.844440 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:54.844632 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.844453 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:54.844632 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.844466 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab025762-9af1-4c25-a0cc-4ab9c0221987-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:54.926304 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.926199 4861 generic.go:356] "Generic (PLEG): container finished" podID="ab025762-9af1-4c25-a0cc-4ab9c0221987" containerID="145ae9daaea2719442e60284123893434d1252b076cf2620bd82f9b919872c02" exitCode=0 Oct 30 16:50:54.926304 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.926260 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" Oct 30 16:50:54.926304 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.926278 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" event={"ID":"ab025762-9af1-4c25-a0cc-4ab9c0221987","Type":"ContainerDied","Data":"145ae9daaea2719442e60284123893434d1252b076cf2620bd82f9b919872c02"} Oct 30 16:50:54.926576 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.926343 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5" event={"ID":"ab025762-9af1-4c25-a0cc-4ab9c0221987","Type":"ContainerDied","Data":"0f4ba2809fc77f0ea57e5fd21aa30f8da40affb8013a3caed2fc60fd09e64639"} Oct 30 16:50:54.926576 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.926379 4861 scope.go:122] "RemoveContainer" containerID="145ae9daaea2719442e60284123893434d1252b076cf2620bd82f9b919872c02" Oct 30 16:50:54.938363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.938344 4861 scope.go:122] "RemoveContainer" containerID="787fff535d36a91672b7d646daf82f9e95b75a9c6d6d43948e30a7472849d818" Oct 30 16:50:54.948107 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.948081 4861 scope.go:122] "RemoveContainer" containerID="145ae9daaea2719442e60284123893434d1252b076cf2620bd82f9b919872c02" Oct 30 16:50:54.948426 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:50:54.948406 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145ae9daaea2719442e60284123893434d1252b076cf2620bd82f9b919872c02\": container with ID starting with 145ae9daaea2719442e60284123893434d1252b076cf2620bd82f9b919872c02 not found: ID does not exist" containerID="145ae9daaea2719442e60284123893434d1252b076cf2620bd82f9b919872c02" Oct 30 16:50:54.948511 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.948434 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145ae9daaea2719442e60284123893434d1252b076cf2620bd82f9b919872c02"} err="failed to get container status \"145ae9daaea2719442e60284123893434d1252b076cf2620bd82f9b919872c02\": rpc error: code = NotFound desc = could not find container \"145ae9daaea2719442e60284123893434d1252b076cf2620bd82f9b919872c02\": container with ID starting with 145ae9daaea2719442e60284123893434d1252b076cf2620bd82f9b919872c02 not found: ID does not exist" Oct 30 16:50:54.948511 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.948451 4861 scope.go:122] "RemoveContainer" containerID="787fff535d36a91672b7d646daf82f9e95b75a9c6d6d43948e30a7472849d818" Oct 30 16:50:54.948773 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:50:54.948752 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"787fff535d36a91672b7d646daf82f9e95b75a9c6d6d43948e30a7472849d818\": container with ID starting with 787fff535d36a91672b7d646daf82f9e95b75a9c6d6d43948e30a7472849d818 not found: ID does not exist" containerID="787fff535d36a91672b7d646daf82f9e95b75a9c6d6d43948e30a7472849d818" Oct 30 16:50:54.948822 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.948781 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787fff535d36a91672b7d646daf82f9e95b75a9c6d6d43948e30a7472849d818"} err="failed to get container status \"787fff535d36a91672b7d646daf82f9e95b75a9c6d6d43948e30a7472849d818\": rpc error: code = NotFound desc = could not find container \"787fff535d36a91672b7d646daf82f9e95b75a9c6d6d43948e30a7472849d818\": container with ID starting with 787fff535d36a91672b7d646daf82f9e95b75a9c6d6d43948e30a7472849d818 not found: ID does not exist" Oct 30 16:50:54.980513 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.980493 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5"] Oct 30 16:50:54.993206 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:54.993186 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-2e9431-openshift-default-8668c4c9bf-g9pj5"] Oct 30 16:50:55.295425 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.295393 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:50:55.351635 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.351602 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-workload-socket\") pod \"e3c4a663-826a-4d68-9208-e16cb76839b5\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " Oct 30 16:50:55.351786 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.351675 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-workload-certs\") pod \"e3c4a663-826a-4d68-9208-e16cb76839b5\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " Oct 30 16:50:55.351786 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.351739 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-data\") pod \"e3c4a663-826a-4d68-9208-e16cb76839b5\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " Oct 30 16:50:55.351950 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.351789 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-credential-socket\") pod \"e3c4a663-826a-4d68-9208-e16cb76839b5\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " Oct 30 16:50:55.351950 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.351816 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-podinfo\") pod \"e3c4a663-826a-4d68-9208-e16cb76839b5\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " Oct 30 16:50:55.351950 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.351851 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-token\" (UniqueName: \"kubernetes.io/projected/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-token\") pod \"e3c4a663-826a-4d68-9208-e16cb76839b5\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " Oct 30 16:50:55.351950 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.351877 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-envoy\") pod \"e3c4a663-826a-4d68-9208-e16cb76839b5\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " Oct 30 16:50:55.352157 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.351946 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-workload-socket" pod "e3c4a663-826a-4d68-9208-e16cb76839b5" (UID: "e3c4a663-826a-4d68-9208-e16cb76839b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:55.352157 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.352000 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c4a663-826a-4d68-9208-e16cb76839b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c4a663-826a-4d68-9208-e16cb76839b5-istiod-ca-cert\") pod \"e3c4a663-826a-4d68-9208-e16cb76839b5\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " Oct 30 16:50:55.352157 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.352016 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-workload-certs" pod "e3c4a663-826a-4d68-9208-e16cb76839b5" (UID: "e3c4a663-826a-4d68-9208-e16cb76839b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:55.352303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.352186 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-credential-socket" pod "e3c4a663-826a-4d68-9208-e16cb76839b5" (UID: "e3c4a663-826a-4d68-9208-e16cb76839b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:55.352303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.352236 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c4a663-826a-4d68-9208-e16cb76839b5-kube-api-access-bqscm\" (UniqueName: \"kubernetes.io/projected/e3c4a663-826a-4d68-9208-e16cb76839b5-kube-api-access-bqscm\") pod \"e3c4a663-826a-4d68-9208-e16cb76839b5\" (UID: \"e3c4a663-826a-4d68-9208-e16cb76839b5\") " Oct 30 16:50:55.352606 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.352583 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-data" pod "e3c4a663-826a-4d68-9208-e16cb76839b5" (UID: "e3c4a663-826a-4d68-9208-e16cb76839b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:55.354023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.352832 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c4a663-826a-4d68-9208-e16cb76839b5-istiod-ca-cert" pod "e3c4a663-826a-4d68-9208-e16cb76839b5" (UID: "e3c4a663-826a-4d68-9208-e16cb76839b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:55.354023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.352936 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:55.354023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.352953 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:55.354023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.352966 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:55.354023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.352980 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:55.354023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.352991 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c4a663-826a-4d68-9208-e16cb76839b5-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:55.355325 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.355300 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c4a663-826a-4d68-9208-e16cb76839b5-kube-api-access-bqscm" pod "e3c4a663-826a-4d68-9208-e16cb76839b5" (UID: "e3c4a663-826a-4d68-9208-e16cb76839b5"). InnerVolumeSpecName "kube-api-access-bqscm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:55.355439 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.355391 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-token" pod "e3c4a663-826a-4d68-9208-e16cb76839b5" (UID: "e3c4a663-826a-4d68-9208-e16cb76839b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:55.355510 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.355437 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-envoy" pod "e3c4a663-826a-4d68-9208-e16cb76839b5" (UID: "e3c4a663-826a-4d68-9208-e16cb76839b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:55.355560 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.355511 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-podinfo" pod "e3c4a663-826a-4d68-9208-e16cb76839b5" (UID: "e3c4a663-826a-4d68-9208-e16cb76839b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:55.453860 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.453833 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqscm\" (UniqueName: \"kubernetes.io/projected/e3c4a663-826a-4d68-9208-e16cb76839b5-kube-api-access-bqscm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:55.453860 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.453857 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:55.453860 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.453866 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:55.454054 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.453875 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c4a663-826a-4d68-9208-e16cb76839b5-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:55.807807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.807773 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab025762-9af1-4c25-a0cc-4ab9c0221987" path="/var/lib/kubelet/pods/ab025762-9af1-4c25-a0cc-4ab9c0221987/volumes" Oct 30 16:50:55.934605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.934565 4861 generic.go:356] "Generic (PLEG): container finished" podID="e3c4a663-826a-4d68-9208-e16cb76839b5" containerID="8f9ca42bd98cf870601928f2955f9e5320a291767a6df1bd19505c5379cedb8f" exitCode=0 Oct 30 16:50:55.934827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.934623 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" Oct 30 16:50:55.934827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.934641 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" event={"ID":"e3c4a663-826a-4d68-9208-e16cb76839b5","Type":"ContainerDied","Data":"8f9ca42bd98cf870601928f2955f9e5320a291767a6df1bd19505c5379cedb8f"} Oct 30 16:50:55.934827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.934701 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc" event={"ID":"e3c4a663-826a-4d68-9208-e16cb76839b5","Type":"ContainerDied","Data":"48d6c575f98bc8e70c1372fde46e465a7f738b894631c478522c347c0ff361da"} Oct 30 16:50:55.934827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.934754 4861 scope.go:122] "RemoveContainer" containerID="8f9ca42bd98cf870601928f2955f9e5320a291767a6df1bd19505c5379cedb8f" Oct 30 16:50:55.945790 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.945775 4861 scope.go:122] "RemoveContainer" containerID="6fbed1c15fbce1d4d93b5dde9ac59029a7b9277a09599863fa88ac2558bb4886" Oct 30 16:50:55.954891 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.954870 4861 scope.go:122] "RemoveContainer" containerID="8f9ca42bd98cf870601928f2955f9e5320a291767a6df1bd19505c5379cedb8f" Oct 30 16:50:55.955178 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:50:55.955159 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9ca42bd98cf870601928f2955f9e5320a291767a6df1bd19505c5379cedb8f\": container with ID starting with 8f9ca42bd98cf870601928f2955f9e5320a291767a6df1bd19505c5379cedb8f not found: ID does not exist" containerID="8f9ca42bd98cf870601928f2955f9e5320a291767a6df1bd19505c5379cedb8f" Oct 30 16:50:55.955245 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.955187 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9ca42bd98cf870601928f2955f9e5320a291767a6df1bd19505c5379cedb8f"} err="failed to get container status \"8f9ca42bd98cf870601928f2955f9e5320a291767a6df1bd19505c5379cedb8f\": rpc error: code = NotFound desc = could not find container \"8f9ca42bd98cf870601928f2955f9e5320a291767a6df1bd19505c5379cedb8f\": container with ID starting with 8f9ca42bd98cf870601928f2955f9e5320a291767a6df1bd19505c5379cedb8f not found: ID does not exist" Oct 30 16:50:55.955245 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.955209 4861 scope.go:122] "RemoveContainer" containerID="6fbed1c15fbce1d4d93b5dde9ac59029a7b9277a09599863fa88ac2558bb4886" Oct 30 16:50:55.955490 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:50:55.955470 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fbed1c15fbce1d4d93b5dde9ac59029a7b9277a09599863fa88ac2558bb4886\": container with ID starting with 6fbed1c15fbce1d4d93b5dde9ac59029a7b9277a09599863fa88ac2558bb4886 not found: ID does not exist" containerID="6fbed1c15fbce1d4d93b5dde9ac59029a7b9277a09599863fa88ac2558bb4886" Oct 30 16:50:55.955533 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.955497 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbed1c15fbce1d4d93b5dde9ac59029a7b9277a09599863fa88ac2558bb4886"} err="failed to get container status \"6fbed1c15fbce1d4d93b5dde9ac59029a7b9277a09599863fa88ac2558bb4886\": rpc error: code = NotFound desc = could not find container \"6fbed1c15fbce1d4d93b5dde9ac59029a7b9277a09599863fa88ac2558bb4886\": container with ID starting with 6fbed1c15fbce1d4d93b5dde9ac59029a7b9277a09599863fa88ac2558bb4886 not found: ID does not exist" Oct 30 16:50:55.991170 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:55.991148 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc"] Oct 30 16:50:56.005052 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:56.005026 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-504442-openshift-default-c6bcbb94f-mtjbc"] Oct 30 16:50:57.403223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.403170 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:50:57.477194 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.477162 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-credential-socket\") pod \"9db4fb72-7b88-4237-8c83-38f14f803caa\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " Oct 30 16:50:57.477357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.477224 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-workload-certs\") pod \"9db4fb72-7b88-4237-8c83-38f14f803caa\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " Oct 30 16:50:57.477357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.477338 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-token\" (UniqueName: \"kubernetes.io/projected/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-token\") pod \"9db4fb72-7b88-4237-8c83-38f14f803caa\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " Oct 30 16:50:57.477436 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.477424 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-envoy\") pod \"9db4fb72-7b88-4237-8c83-38f14f803caa\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " Oct 30 16:50:57.477468 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.477456 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db4fb72-7b88-4237-8c83-38f14f803caa-kube-api-access-h2cn6\" (UniqueName: \"kubernetes.io/projected/9db4fb72-7b88-4237-8c83-38f14f803caa-kube-api-access-h2cn6\") pod \"9db4fb72-7b88-4237-8c83-38f14f803caa\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " Oct 30 16:50:57.477518 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.477487 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-credential-socket" pod "9db4fb72-7b88-4237-8c83-38f14f803caa" (UID: "9db4fb72-7b88-4237-8c83-38f14f803caa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:57.477569 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.477557 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-workload-socket\") pod \"9db4fb72-7b88-4237-8c83-38f14f803caa\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " Oct 30 16:50:57.477607 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.477575 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-workload-certs" pod "9db4fb72-7b88-4237-8c83-38f14f803caa" (UID: "9db4fb72-7b88-4237-8c83-38f14f803caa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:57.477641 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.477631 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db4fb72-7b88-4237-8c83-38f14f803caa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db4fb72-7b88-4237-8c83-38f14f803caa-istiod-ca-cert\") pod \"9db4fb72-7b88-4237-8c83-38f14f803caa\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " Oct 30 16:50:57.478455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.477688 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-podinfo\") pod \"9db4fb72-7b88-4237-8c83-38f14f803caa\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " Oct 30 16:50:57.478455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.477798 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-data\") pod \"9db4fb72-7b88-4237-8c83-38f14f803caa\" (UID: \"9db4fb72-7b88-4237-8c83-38f14f803caa\") " Oct 30 16:50:57.478455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.477944 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-workload-socket" pod "9db4fb72-7b88-4237-8c83-38f14f803caa" (UID: "9db4fb72-7b88-4237-8c83-38f14f803caa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:57.478455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.478370 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-data" pod "9db4fb72-7b88-4237-8c83-38f14f803caa" (UID: "9db4fb72-7b88-4237-8c83-38f14f803caa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:57.478455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.478393 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:57.478455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.478415 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:57.478455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.478432 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:57.478951 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.478545 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db4fb72-7b88-4237-8c83-38f14f803caa-istiod-ca-cert" pod "9db4fb72-7b88-4237-8c83-38f14f803caa" (UID: "9db4fb72-7b88-4237-8c83-38f14f803caa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:57.480359 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.480327 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-token" pod "9db4fb72-7b88-4237-8c83-38f14f803caa" (UID: "9db4fb72-7b88-4237-8c83-38f14f803caa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:57.480605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.480582 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-envoy" pod "9db4fb72-7b88-4237-8c83-38f14f803caa" (UID: "9db4fb72-7b88-4237-8c83-38f14f803caa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:57.480905 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.480881 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db4fb72-7b88-4237-8c83-38f14f803caa-kube-api-access-h2cn6" pod "9db4fb72-7b88-4237-8c83-38f14f803caa" (UID: "9db4fb72-7b88-4237-8c83-38f14f803caa"). InnerVolumeSpecName "kube-api-access-h2cn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:57.481015 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.480949 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-podinfo" pod "9db4fb72-7b88-4237-8c83-38f14f803caa" (UID: "9db4fb72-7b88-4237-8c83-38f14f803caa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:57.580082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.580055 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db4fb72-7b88-4237-8c83-38f14f803caa-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:57.580082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.580078 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:57.580082 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.580086 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:57.580298 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.580095 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:57.580298 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.580103 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db4fb72-7b88-4237-8c83-38f14f803caa-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:57.580298 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.580112 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2cn6\" (UniqueName: \"kubernetes.io/projected/9db4fb72-7b88-4237-8c83-38f14f803caa-kube-api-access-h2cn6\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:50:57.807477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.807441 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c4a663-826a-4d68-9208-e16cb76839b5" path="/var/lib/kubelet/pods/e3c4a663-826a-4d68-9208-e16cb76839b5/volumes" Oct 30 16:50:57.952313 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.952277 4861 generic.go:356] "Generic (PLEG): container finished" podID="9db4fb72-7b88-4237-8c83-38f14f803caa" containerID="9ee60eb06b52bc87e0ff7ff965610e4d6f5696bd10a3b8eda36f23cae36f6ff2" exitCode=0 Oct 30 16:50:57.952477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.952335 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" Oct 30 16:50:57.952477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.952365 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" event={"ID":"9db4fb72-7b88-4237-8c83-38f14f803caa","Type":"ContainerDied","Data":"9ee60eb06b52bc87e0ff7ff965610e4d6f5696bd10a3b8eda36f23cae36f6ff2"} Oct 30 16:50:57.952477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.952429 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj" event={"ID":"9db4fb72-7b88-4237-8c83-38f14f803caa","Type":"ContainerDied","Data":"0e193f36c26863e369ebb221958fe77675f81fd5ee8f7f156eff91a9903c84c6"} Oct 30 16:50:57.952477 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.952459 4861 scope.go:122] "RemoveContainer" containerID="9ee60eb06b52bc87e0ff7ff965610e4d6f5696bd10a3b8eda36f23cae36f6ff2" Oct 30 16:50:57.962519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.962494 4861 scope.go:122] "RemoveContainer" containerID="1dfbad26e9dff50ec551681a222dccd3b0708f50681f62c23176ca48295f4457" Oct 30 16:50:57.971684 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.971666 4861 scope.go:122] "RemoveContainer" containerID="9ee60eb06b52bc87e0ff7ff965610e4d6f5696bd10a3b8eda36f23cae36f6ff2" Oct 30 16:50:57.971936 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:50:57.971919 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee60eb06b52bc87e0ff7ff965610e4d6f5696bd10a3b8eda36f23cae36f6ff2\": container with ID starting with 9ee60eb06b52bc87e0ff7ff965610e4d6f5696bd10a3b8eda36f23cae36f6ff2 not found: ID does not exist" containerID="9ee60eb06b52bc87e0ff7ff965610e4d6f5696bd10a3b8eda36f23cae36f6ff2" Oct 30 16:50:57.972002 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.971945 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee60eb06b52bc87e0ff7ff965610e4d6f5696bd10a3b8eda36f23cae36f6ff2"} err="failed to get container status \"9ee60eb06b52bc87e0ff7ff965610e4d6f5696bd10a3b8eda36f23cae36f6ff2\": rpc error: code = NotFound desc = could not find container \"9ee60eb06b52bc87e0ff7ff965610e4d6f5696bd10a3b8eda36f23cae36f6ff2\": container with ID starting with 9ee60eb06b52bc87e0ff7ff965610e4d6f5696bd10a3b8eda36f23cae36f6ff2 not found: ID does not exist" Oct 30 16:50:57.972002 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.971965 4861 scope.go:122] "RemoveContainer" containerID="1dfbad26e9dff50ec551681a222dccd3b0708f50681f62c23176ca48295f4457" Oct 30 16:50:57.972231 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:50:57.972216 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfbad26e9dff50ec551681a222dccd3b0708f50681f62c23176ca48295f4457\": container with ID starting with 1dfbad26e9dff50ec551681a222dccd3b0708f50681f62c23176ca48295f4457 not found: ID does not exist" containerID="1dfbad26e9dff50ec551681a222dccd3b0708f50681f62c23176ca48295f4457" Oct 30 16:50:57.972269 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:57.972247 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfbad26e9dff50ec551681a222dccd3b0708f50681f62c23176ca48295f4457"} err="failed to get container status \"1dfbad26e9dff50ec551681a222dccd3b0708f50681f62c23176ca48295f4457\": rpc error: code = NotFound desc = could not find container \"1dfbad26e9dff50ec551681a222dccd3b0708f50681f62c23176ca48295f4457\": container with ID starting with 1dfbad26e9dff50ec551681a222dccd3b0708f50681f62c23176ca48295f4457 not found: ID does not exist" Oct 30 16:50:58.019434 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:58.019385 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj"] Oct 30 16:50:58.032268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:58.032226 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-72b379-openshift-default-7c44cc87fc-x7lmj"] Oct 30 16:50:59.808308 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:50:59.808275 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db4fb72-7b88-4237-8c83-38f14f803caa" path="/var/lib/kubelet/pods/9db4fb72-7b88-4237-8c83-38f14f803caa/volumes" Oct 30 16:51:00.932195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.932163 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8"] Oct 30 16:51:00.932828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.932801 4861 cpu_manager.go:414] "containerMap: removing container" podUID="9db4fb72-7b88-4237-8c83-38f14f803caa" containerName="istio-proxy" Oct 30 16:51:00.932828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.932823 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db4fb72-7b88-4237-8c83-38f14f803caa" containerName="istio-proxy" Oct 30 16:51:00.932991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.932863 4861 cpu_manager.go:414] "containerMap: removing container" podUID="ab025762-9af1-4c25-a0cc-4ab9c0221987" containerName="istio-proxy" Oct 30 16:51:00.932991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.932869 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab025762-9af1-4c25-a0cc-4ab9c0221987" containerName="istio-proxy" Oct 30 16:51:00.932991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.932879 4861 cpu_manager.go:414] "containerMap: removing container" podUID="331f3eb5-061d-40f1-9ff2-9f1e90e3619c" containerName="maas-api" Oct 30 16:51:00.932991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.932884 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="331f3eb5-061d-40f1-9ff2-9f1e90e3619c" containerName="maas-api" Oct 30 16:51:00.932991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.932897 4861 cpu_manager.go:414] "containerMap: removing container" podUID="ab025762-9af1-4c25-a0cc-4ab9c0221987" containerName="istio-proxy" Oct 30 16:51:00.932991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.932901 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab025762-9af1-4c25-a0cc-4ab9c0221987" containerName="istio-proxy" Oct 30 16:51:00.932991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.932913 4861 cpu_manager.go:414] "containerMap: removing container" podUID="e3c4a663-826a-4d68-9208-e16cb76839b5" containerName="istio-proxy" Oct 30 16:51:00.932991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.932917 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c4a663-826a-4d68-9208-e16cb76839b5" containerName="istio-proxy" Oct 30 16:51:00.932991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.932931 4861 cpu_manager.go:414] "containerMap: removing container" podUID="e3c4a663-826a-4d68-9208-e16cb76839b5" containerName="istio-proxy" Oct 30 16:51:00.932991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.932935 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c4a663-826a-4d68-9208-e16cb76839b5" containerName="istio-proxy" Oct 30 16:51:00.933266 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.933022 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db4fb72-7b88-4237-8c83-38f14f803caa" containerName="istio-proxy" Oct 30 16:51:00.933266 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.933034 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab025762-9af1-4c25-a0cc-4ab9c0221987" containerName="istio-proxy" Oct 30 16:51:00.933266 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.933040 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="331f3eb5-061d-40f1-9ff2-9f1e90e3619c" containerName="maas-api" Oct 30 16:51:00.933266 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.933046 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab025762-9af1-4c25-a0cc-4ab9c0221987" containerName="istio-proxy" Oct 30 16:51:00.933266 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.933053 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c4a663-826a-4d68-9208-e16cb76839b5" containerName="istio-proxy" Oct 30 16:51:00.937960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.937942 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:00.968593 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.968566 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-18519c-openshift-default-dockercfg-nqftk\"" Oct 30 16:51:00.974912 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:00.974874 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8"] Oct 30 16:51:01.017995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.017940 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.018313 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.018277 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.018449 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.018435 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.018590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.018563 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.018784 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.018768 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9951912-7c87-4ed5-8525-073105af8c0c-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.018929 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.018911 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdpzj\" (UniqueName: \"kubernetes.io/projected/f9951912-7c87-4ed5-8525-073105af8c0c-kube-api-access-pdpzj\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.019297 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.019275 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.019521 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.019455 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9951912-7c87-4ed5-8525-073105af8c0c-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.019682 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.019663 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9951912-7c87-4ed5-8525-073105af8c0c-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.121323 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.121292 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.121486 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.121327 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9951912-7c87-4ed5-8525-073105af8c0c-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.121486 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.121445 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9951912-7c87-4ed5-8525-073105af8c0c-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.121616 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.121598 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.121680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.121664 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.121753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.121696 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.121823 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.121752 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.121987 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.121964 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.122072 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.121757 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.122072 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.122037 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.122072 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.122058 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.122247 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.122081 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9951912-7c87-4ed5-8525-073105af8c0c-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.122247 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.122128 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdpzj\" (UniqueName: \"kubernetes.io/projected/f9951912-7c87-4ed5-8525-073105af8c0c-kube-api-access-pdpzj\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.122340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.122255 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9951912-7c87-4ed5-8525-073105af8c0c-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.124293 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.124270 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9951912-7c87-4ed5-8525-073105af8c0c-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.124419 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.124401 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.145939 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.145919 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9951912-7c87-4ed5-8525-073105af8c0c-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.146276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.146255 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdpzj\" (UniqueName: \"kubernetes.io/projected/f9951912-7c87-4ed5-8525-073105af8c0c-kube-api-access-pdpzj\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.252241 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.252217 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:01.584118 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.584084 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8"] Oct 30 16:51:01.585401 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:51:01.585375 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9951912_7c87_4ed5_8525_073105af8c0c.slice/crio-c614e377637a96fb05f68429fb66da708fb48873a380ac52294fd19b69791e37 WatchSource:0}: Error finding container c614e377637a96fb05f68429fb66da708fb48873a380ac52294fd19b69791e37: Status 404 returned error can't find the container with id c614e377637a96fb05f68429fb66da708fb48873a380ac52294fd19b69791e37 Oct 30 16:51:01.587736 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.587690 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:01.587812 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.587785 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:01.587848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:01.587814 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:02.030355 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:02.030320 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" event={"ID":"f9951912-7c87-4ed5-8525-073105af8c0c","Type":"ContainerStarted","Data":"e091d2ff1cd52d94d8c061ef7e6fcdafc82a5cd93add9a69a7409407ba52fd93"} Oct 30 16:51:02.030846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:02.030376 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" event={"ID":"f9951912-7c87-4ed5-8525-073105af8c0c","Type":"ContainerStarted","Data":"c614e377637a96fb05f68429fb66da708fb48873a380ac52294fd19b69791e37"} Oct 30 16:51:02.077673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:02.077622 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" podStartSLOduration=2.077608983 podStartE2EDuration="2.077608983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:02.076698679 +0000 UTC m=+1692.419372718" watchObservedRunningTime="2026-10-30 16:51:02.077608983 +0000 UTC m=+1692.420283077" Oct 30 16:51:02.254162 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:02.254045 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:02.258746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:02.256878 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 16:51:02.258746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:02.256940 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" podUID="f9951912-7c87-4ed5-8525-073105af8c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 16:51:03.252634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:03.252590 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 16:51:03.253099 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:03.252645 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" podUID="f9951912-7c87-4ed5-8525-073105af8c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 16:51:03.536674 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:03.536580 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v"] Oct 30 16:51:03.589126 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:03.585893 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt"] Oct 30 16:51:04.252659 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:04.252624 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 16:51:04.254201 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:04.254167 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" podUID="f9951912-7c87-4ed5-8525-073105af8c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 16:51:05.253365 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:05.253325 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 16:51:05.253783 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:05.253394 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" podUID="f9951912-7c87-4ed5-8525-073105af8c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 16:51:06.252738 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:06.252695 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 16:51:06.252901 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:06.252764 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" podUID="f9951912-7c87-4ed5-8525-073105af8c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 16:51:07.253400 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:07.253365 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 16:51:07.253820 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:07.253429 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" podUID="f9951912-7c87-4ed5-8525-073105af8c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 16:51:08.252659 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.252625 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 16:51:08.252889 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.252681 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" podUID="f9951912-7c87-4ed5-8525-073105af8c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 16:51:08.363702 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.363662 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl"] Oct 30 16:51:08.364194 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.363933 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" podUID="dac9dbe1-f2bc-4feb-b107-b94cca178634" containerName="maas-api" containerID="cri-o://a66ae9b7e12745f797b6d15043b8d1a1a28ae50ab5f6a2211f6a17c6cca26cfa" gracePeriod=30 Oct 30 16:51:08.537470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.537387 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" podUID="1e318052-ec4f-406a-97ff-a1a99b698595" containerName="payload-processing" containerID="cri-o://aab121455eadc188746bc80841c849f20eef335f89b665e6611d7aa61bd87b17" gracePeriod=25 Oct 30 16:51:08.586700 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.586629 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" podUID="63d93220-40d7-4b64-ba7b-9764ad028511" containerName="payload-pre-processing" containerID="cri-o://267bf3da7f151c1874fa513f05fa42882b7e6edbbd813558b73dd9c1b4f1baad" gracePeriod=25 Oct 30 16:51:08.752319 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.752044 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:51:08.893328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.892364 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dac9dbe1-f2bc-4feb-b107-b94cca178634-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac9dbe1-f2bc-4feb-b107-b94cca178634-metrics-certs\") pod \"dac9dbe1-f2bc-4feb-b107-b94cca178634\" (UID: \"dac9dbe1-f2bc-4feb-b107-b94cca178634\") " Oct 30 16:51:08.893328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.892522 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac9dbe1-f2bc-4feb-b107-b94cca178634-kube-api-access-qgnvl\" (UniqueName: \"kubernetes.io/projected/dac9dbe1-f2bc-4feb-b107-b94cca178634-kube-api-access-qgnvl\") pod \"dac9dbe1-f2bc-4feb-b107-b94cca178634\" (UID: \"dac9dbe1-f2bc-4feb-b107-b94cca178634\") " Oct 30 16:51:08.893328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.892557 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dac9dbe1-f2bc-4feb-b107-b94cca178634-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dac9dbe1-f2bc-4feb-b107-b94cca178634-maas-api-tls\") pod \"dac9dbe1-f2bc-4feb-b107-b94cca178634\" (UID: \"dac9dbe1-f2bc-4feb-b107-b94cca178634\") " Oct 30 16:51:08.893328 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.892648 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dac9dbe1-f2bc-4feb-b107-b94cca178634-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac9dbe1-f2bc-4feb-b107-b94cca178634-openshift-service-ca\") pod \"dac9dbe1-f2bc-4feb-b107-b94cca178634\" (UID: \"dac9dbe1-f2bc-4feb-b107-b94cca178634\") " Oct 30 16:51:08.894849 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.894812 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac9dbe1-f2bc-4feb-b107-b94cca178634-openshift-service-ca" pod "dac9dbe1-f2bc-4feb-b107-b94cca178634" (UID: "dac9dbe1-f2bc-4feb-b107-b94cca178634"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:08.900948 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.900900 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac9dbe1-f2bc-4feb-b107-b94cca178634-kube-api-access-qgnvl" pod "dac9dbe1-f2bc-4feb-b107-b94cca178634" (UID: "dac9dbe1-f2bc-4feb-b107-b94cca178634"). InnerVolumeSpecName "kube-api-access-qgnvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:08.903020 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.902976 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dac9dbe1-f2bc-4feb-b107-b94cca178634-metrics-certs" pod "dac9dbe1-f2bc-4feb-b107-b94cca178634" (UID: "dac9dbe1-f2bc-4feb-b107-b94cca178634"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:08.906942 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.906906 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dac9dbe1-f2bc-4feb-b107-b94cca178634-maas-api-tls" pod "dac9dbe1-f2bc-4feb-b107-b94cca178634" (UID: "dac9dbe1-f2bc-4feb-b107-b94cca178634"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:08.995130 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.995097 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac9dbe1-f2bc-4feb-b107-b94cca178634-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:08.995287 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.995149 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac9dbe1-f2bc-4feb-b107-b94cca178634-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:08.995287 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.995167 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgnvl\" (UniqueName: \"kubernetes.io/projected/dac9dbe1-f2bc-4feb-b107-b94cca178634-kube-api-access-qgnvl\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:08.995287 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:08.995179 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dac9dbe1-f2bc-4feb-b107-b94cca178634-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:09.011060 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.010689 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" Oct 30 16:51:09.080592 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.080528 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" Oct 30 16:51:09.081016 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.080888 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" event={"ID":"dac9dbe1-f2bc-4feb-b107-b94cca178634","Type":"ContainerDied","Data":"a66ae9b7e12745f797b6d15043b8d1a1a28ae50ab5f6a2211f6a17c6cca26cfa"} Oct 30 16:51:09.081251 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.081175 4861 scope.go:122] "RemoveContainer" containerID="a66ae9b7e12745f797b6d15043b8d1a1a28ae50ab5f6a2211f6a17c6cca26cfa" Oct 30 16:51:09.084442 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.084331 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" Oct 30 16:51:09.094771 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.080401 4861 generic.go:356] "Generic (PLEG): container finished" podID="dac9dbe1-f2bc-4feb-b107-b94cca178634" containerID="a66ae9b7e12745f797b6d15043b8d1a1a28ae50ab5f6a2211f6a17c6cca26cfa" exitCode=0 Oct 30 16:51:09.095038 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.094973 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl" event={"ID":"dac9dbe1-f2bc-4feb-b107-b94cca178634","Type":"ContainerDied","Data":"595611a5e22de2200a3fbe12e012f69d1be8439555e0872f14a47d5141f87afa"} Oct 30 16:51:09.096064 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.095967 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e318052-ec4f-406a-97ff-a1a99b698595-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e318052-ec4f-406a-97ff-a1a99b698595-plugins-config-volume\") pod \"1e318052-ec4f-406a-97ff-a1a99b698595\" (UID: \"1e318052-ec4f-406a-97ff-a1a99b698595\") " Oct 30 16:51:09.096980 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.096015 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d93220-40d7-4b64-ba7b-9764ad028511-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d93220-40d7-4b64-ba7b-9764ad028511-plugins-config-volume\") pod \"63d93220-40d7-4b64-ba7b-9764ad028511\" (UID: \"63d93220-40d7-4b64-ba7b-9764ad028511\") " Oct 30 16:51:09.096980 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.096206 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e318052-ec4f-406a-97ff-a1a99b698595-kube-api-access-4p6gj\" (UniqueName: \"kubernetes.io/projected/1e318052-ec4f-406a-97ff-a1a99b698595-kube-api-access-4p6gj\") pod \"1e318052-ec4f-406a-97ff-a1a99b698595\" (UID: \"1e318052-ec4f-406a-97ff-a1a99b698595\") " Oct 30 16:51:09.096980 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.096247 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d93220-40d7-4b64-ba7b-9764ad028511-kube-api-access-q9w9l\" (UniqueName: \"kubernetes.io/projected/63d93220-40d7-4b64-ba7b-9764ad028511-kube-api-access-q9w9l\") pod \"63d93220-40d7-4b64-ba7b-9764ad028511\" (UID: \"63d93220-40d7-4b64-ba7b-9764ad028511\") " Oct 30 16:51:09.098061 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.097796 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d93220-40d7-4b64-ba7b-9764ad028511-plugins-config-volume" pod "63d93220-40d7-4b64-ba7b-9764ad028511" (UID: "63d93220-40d7-4b64-ba7b-9764ad028511"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:09.099246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.099213 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e318052-ec4f-406a-97ff-a1a99b698595-plugins-config-volume" pod "1e318052-ec4f-406a-97ff-a1a99b698595" (UID: "1e318052-ec4f-406a-97ff-a1a99b698595"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:09.102745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.101738 4861 generic.go:356] "Generic (PLEG): container finished" podID="63d93220-40d7-4b64-ba7b-9764ad028511" containerID="267bf3da7f151c1874fa513f05fa42882b7e6edbbd813558b73dd9c1b4f1baad" exitCode=0 Oct 30 16:51:09.102745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.101932 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" event={"ID":"63d93220-40d7-4b64-ba7b-9764ad028511","Type":"ContainerDied","Data":"267bf3da7f151c1874fa513f05fa42882b7e6edbbd813558b73dd9c1b4f1baad"} Oct 30 16:51:09.102745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.101976 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" event={"ID":"63d93220-40d7-4b64-ba7b-9764ad028511","Type":"ContainerDied","Data":"784251da3b8ca73f463db325a8c21fa9bce05c3ec0a8b4e571a0f329e41e62d3"} Oct 30 16:51:09.102745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.102051 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt" Oct 30 16:51:09.118801 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.104855 4861 generic.go:356] "Generic (PLEG): container finished" podID="1e318052-ec4f-406a-97ff-a1a99b698595" containerID="aab121455eadc188746bc80841c849f20eef335f89b665e6611d7aa61bd87b17" exitCode=0 Oct 30 16:51:09.118801 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.104941 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" event={"ID":"1e318052-ec4f-406a-97ff-a1a99b698595","Type":"ContainerDied","Data":"aab121455eadc188746bc80841c849f20eef335f89b665e6611d7aa61bd87b17"} Oct 30 16:51:09.118801 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.105049 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" event={"ID":"1e318052-ec4f-406a-97ff-a1a99b698595","Type":"ContainerDied","Data":"2b76353949d3a947677ecd6c8e66bc4128994c928bc82e57a4b2107cd6850c5f"} Oct 30 16:51:09.118801 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.105143 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v" Oct 30 16:51:09.122830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.122791 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e318052-ec4f-406a-97ff-a1a99b698595-kube-api-access-4p6gj" pod "1e318052-ec4f-406a-97ff-a1a99b698595" (UID: "1e318052-ec4f-406a-97ff-a1a99b698595"). InnerVolumeSpecName "kube-api-access-4p6gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:09.125455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.125440 4861 scope.go:122] "RemoveContainer" containerID="a66ae9b7e12745f797b6d15043b8d1a1a28ae50ab5f6a2211f6a17c6cca26cfa" Oct 30 16:51:09.125800 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:51:09.125771 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a66ae9b7e12745f797b6d15043b8d1a1a28ae50ab5f6a2211f6a17c6cca26cfa\": container with ID starting with a66ae9b7e12745f797b6d15043b8d1a1a28ae50ab5f6a2211f6a17c6cca26cfa not found: ID does not exist" containerID="a66ae9b7e12745f797b6d15043b8d1a1a28ae50ab5f6a2211f6a17c6cca26cfa" Oct 30 16:51:09.125883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.125814 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66ae9b7e12745f797b6d15043b8d1a1a28ae50ab5f6a2211f6a17c6cca26cfa"} err="failed to get container status \"a66ae9b7e12745f797b6d15043b8d1a1a28ae50ab5f6a2211f6a17c6cca26cfa\": rpc error: code = NotFound desc = could not find container \"a66ae9b7e12745f797b6d15043b8d1a1a28ae50ab5f6a2211f6a17c6cca26cfa\": container with ID starting with a66ae9b7e12745f797b6d15043b8d1a1a28ae50ab5f6a2211f6a17c6cca26cfa not found: ID does not exist" Oct 30 16:51:09.125883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.125837 4861 scope.go:122] "RemoveContainer" containerID="267bf3da7f151c1874fa513f05fa42882b7e6edbbd813558b73dd9c1b4f1baad" Oct 30 16:51:09.139992 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.139533 4861 scope.go:122] "RemoveContainer" containerID="267bf3da7f151c1874fa513f05fa42882b7e6edbbd813558b73dd9c1b4f1baad" Oct 30 16:51:09.140092 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:51:09.140063 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"267bf3da7f151c1874fa513f05fa42882b7e6edbbd813558b73dd9c1b4f1baad\": container with ID starting with 267bf3da7f151c1874fa513f05fa42882b7e6edbbd813558b73dd9c1b4f1baad not found: ID does not exist" containerID="267bf3da7f151c1874fa513f05fa42882b7e6edbbd813558b73dd9c1b4f1baad" Oct 30 16:51:09.140178 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.140106 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"267bf3da7f151c1874fa513f05fa42882b7e6edbbd813558b73dd9c1b4f1baad"} err="failed to get container status \"267bf3da7f151c1874fa513f05fa42882b7e6edbbd813558b73dd9c1b4f1baad\": rpc error: code = NotFound desc = could not find container \"267bf3da7f151c1874fa513f05fa42882b7e6edbbd813558b73dd9c1b4f1baad\": container with ID starting with 267bf3da7f151c1874fa513f05fa42882b7e6edbbd813558b73dd9c1b4f1baad not found: ID does not exist" Oct 30 16:51:09.140178 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.140130 4861 scope.go:122] "RemoveContainer" containerID="aab121455eadc188746bc80841c849f20eef335f89b665e6611d7aa61bd87b17" Oct 30 16:51:09.141837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.141799 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d93220-40d7-4b64-ba7b-9764ad028511-kube-api-access-q9w9l" pod "63d93220-40d7-4b64-ba7b-9764ad028511" (UID: "63d93220-40d7-4b64-ba7b-9764ad028511"). InnerVolumeSpecName "kube-api-access-q9w9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:09.179392 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.179262 4861 scope.go:122] "RemoveContainer" containerID="aab121455eadc188746bc80841c849f20eef335f89b665e6611d7aa61bd87b17" Oct 30 16:51:09.180305 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:51:09.180223 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aab121455eadc188746bc80841c849f20eef335f89b665e6611d7aa61bd87b17\": container with ID starting with aab121455eadc188746bc80841c849f20eef335f89b665e6611d7aa61bd87b17 not found: ID does not exist" containerID="aab121455eadc188746bc80841c849f20eef335f89b665e6611d7aa61bd87b17" Oct 30 16:51:09.180305 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.180265 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aab121455eadc188746bc80841c849f20eef335f89b665e6611d7aa61bd87b17"} err="failed to get container status \"aab121455eadc188746bc80841c849f20eef335f89b665e6611d7aa61bd87b17\": rpc error: code = NotFound desc = could not find container \"aab121455eadc188746bc80841c849f20eef335f89b665e6611d7aa61bd87b17\": container with ID starting with aab121455eadc188746bc80841c849f20eef335f89b665e6611d7aa61bd87b17 not found: ID does not exist" Oct 30 16:51:09.199587 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.199553 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e318052-ec4f-406a-97ff-a1a99b698595-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:09.199758 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.199593 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63d93220-40d7-4b64-ba7b-9764ad028511-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:09.199758 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.199611 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p6gj\" (UniqueName: \"kubernetes.io/projected/1e318052-ec4f-406a-97ff-a1a99b698595-kube-api-access-4p6gj\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:09.199758 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.199626 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9w9l\" (UniqueName: \"kubernetes.io/projected/63d93220-40d7-4b64-ba7b-9764ad028511-kube-api-access-q9w9l\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:09.222380 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.222350 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl"] Oct 30 16:51:09.238101 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.238066 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c408f1-5879cdb5d9-ks4hl"] Oct 30 16:51:09.487889 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.487859 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v"] Oct 30 16:51:09.494667 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.494642 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c408f1-66f7b58bb-lfx4v"] Oct 30 16:51:09.530885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.530855 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt"] Oct 30 16:51:09.536105 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.536079 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c408f1-bdfc68655-p5pjt"] Oct 30 16:51:09.809287 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.809249 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e318052-ec4f-406a-97ff-a1a99b698595" path="/var/lib/kubelet/pods/1e318052-ec4f-406a-97ff-a1a99b698595/volumes" Oct 30 16:51:09.809884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.809853 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d93220-40d7-4b64-ba7b-9764ad028511" path="/var/lib/kubelet/pods/63d93220-40d7-4b64-ba7b-9764ad028511/volumes" Oct 30 16:51:09.810478 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:09.810456 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac9dbe1-f2bc-4feb-b107-b94cca178634" path="/var/lib/kubelet/pods/dac9dbe1-f2bc-4feb-b107-b94cca178634/volumes" Oct 30 16:51:10.293895 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:10.293855 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:51:10.294075 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:10.293919 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" podUID="f9951912-7c87-4ed5-8525-073105af8c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:51:10.708992 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:10.708905 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:10.709565 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:10.709280 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:10.710068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:10.710048 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:51:11.549924 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:11.549891 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl"] Oct 30 16:51:11.550174 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:11.550150 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" podUID="5fa04564-6eb3-4184-b9e2-3d64ac6752ad" containerName="maas-api" containerID="cri-o://759c27d3db6949410211857f0c4ef5b3f1a7ee69b1fcb5f4a30f1b89d02f7958" gracePeriod=30 Oct 30 16:51:11.737122 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:11.733326 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz"] Oct 30 16:51:11.737122 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:11.733562 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" podUID="1916010e-81c5-4927-af79-d4d9b343ace4" containerName="payload-processing" containerID="cri-o://bb8ac86aa8771f758ed4b7b8ed8b98045642f933fae2c33bd34eca5114822b17" gracePeriod=30 Oct 30 16:51:11.762287 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:11.762264 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz"] Oct 30 16:51:11.762475 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:11.762457 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" podUID="d3e15e44-d162-4dc6-9c52-8ac1be42411f" containerName="payload-pre-processing" containerID="cri-o://6b6d163e16d031dd34af88c7648301409573ea3b3d6482f8f6a04ecb34696f65" gracePeriod=30 Oct 30 16:51:11.905854 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:11.905824 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:51:11.937277 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:11.937237 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-metrics-certs\") pod \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\" (UID: \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\") " Oct 30 16:51:11.937401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:11.937314 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-maas-api-tls\") pod \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\" (UID: \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\") " Oct 30 16:51:11.937401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:11.937391 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-openshift-service-ca\") pod \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\" (UID: \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\") " Oct 30 16:51:11.937631 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:11.937441 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-kube-api-access-n8qvd\" (UniqueName: \"kubernetes.io/projected/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-kube-api-access-n8qvd\") pod \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\" (UID: \"5fa04564-6eb3-4184-b9e2-3d64ac6752ad\") " Oct 30 16:51:11.939223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:11.939179 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-openshift-service-ca" pod "5fa04564-6eb3-4184-b9e2-3d64ac6752ad" (UID: "5fa04564-6eb3-4184-b9e2-3d64ac6752ad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:11.939578 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:11.939550 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:11.941155 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:11.941121 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-metrics-certs" pod "5fa04564-6eb3-4184-b9e2-3d64ac6752ad" (UID: "5fa04564-6eb3-4184-b9e2-3d64ac6752ad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:11.941702 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:11.941675 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-maas-api-tls" pod "5fa04564-6eb3-4184-b9e2-3d64ac6752ad" (UID: "5fa04564-6eb3-4184-b9e2-3d64ac6752ad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:11.942425 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:11.942389 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-kube-api-access-n8qvd" pod "5fa04564-6eb3-4184-b9e2-3d64ac6752ad" (UID: "5fa04564-6eb3-4184-b9e2-3d64ac6752ad"). InnerVolumeSpecName "kube-api-access-n8qvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:12.029793 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.029766 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" Oct 30 16:51:12.041002 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.040974 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:12.041002 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.041001 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:12.041178 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.041014 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8qvd\" (UniqueName: \"kubernetes.io/projected/5fa04564-6eb3-4184-b9e2-3d64ac6752ad-kube-api-access-n8qvd\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:12.095808 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.094928 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" Oct 30 16:51:12.142268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.142230 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e15e44-d162-4dc6-9c52-8ac1be42411f-kube-api-access-s7rr6\" (UniqueName: \"kubernetes.io/projected/d3e15e44-d162-4dc6-9c52-8ac1be42411f-kube-api-access-s7rr6\") pod \"d3e15e44-d162-4dc6-9c52-8ac1be42411f\" (UID: \"d3e15e44-d162-4dc6-9c52-8ac1be42411f\") " Oct 30 16:51:12.142471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.142297 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1916010e-81c5-4927-af79-d4d9b343ace4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1916010e-81c5-4927-af79-d4d9b343ace4-plugins-config-volume\") pod \"1916010e-81c5-4927-af79-d4d9b343ace4\" (UID: \"1916010e-81c5-4927-af79-d4d9b343ace4\") " Oct 30 16:51:12.142471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.142340 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1916010e-81c5-4927-af79-d4d9b343ace4-kube-api-access-7phvm\" (UniqueName: \"kubernetes.io/projected/1916010e-81c5-4927-af79-d4d9b343ace4-kube-api-access-7phvm\") pod \"1916010e-81c5-4927-af79-d4d9b343ace4\" (UID: \"1916010e-81c5-4927-af79-d4d9b343ace4\") " Oct 30 16:51:12.142471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.142400 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e15e44-d162-4dc6-9c52-8ac1be42411f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e15e44-d162-4dc6-9c52-8ac1be42411f-plugins-config-volume\") pod \"d3e15e44-d162-4dc6-9c52-8ac1be42411f\" (UID: \"d3e15e44-d162-4dc6-9c52-8ac1be42411f\") " Oct 30 16:51:12.147598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.146067 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1916010e-81c5-4927-af79-d4d9b343ace4-plugins-config-volume" pod "1916010e-81c5-4927-af79-d4d9b343ace4" (UID: "1916010e-81c5-4927-af79-d4d9b343ace4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:12.147598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.146521 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e15e44-d162-4dc6-9c52-8ac1be42411f-plugins-config-volume" pod "d3e15e44-d162-4dc6-9c52-8ac1be42411f" (UID: "d3e15e44-d162-4dc6-9c52-8ac1be42411f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:12.171982 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.167986 4861 generic.go:356] "Generic (PLEG): container finished" podID="d3e15e44-d162-4dc6-9c52-8ac1be42411f" containerID="6b6d163e16d031dd34af88c7648301409573ea3b3d6482f8f6a04ecb34696f65" exitCode=0 Oct 30 16:51:12.171982 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.168137 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" event={"ID":"d3e15e44-d162-4dc6-9c52-8ac1be42411f","Type":"ContainerDied","Data":"6b6d163e16d031dd34af88c7648301409573ea3b3d6482f8f6a04ecb34696f65"} Oct 30 16:51:12.171982 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.168190 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" event={"ID":"d3e15e44-d162-4dc6-9c52-8ac1be42411f","Type":"ContainerDied","Data":"b34a55bce64f876303a75671470f70646ba66d1c4f2d89eeed74d27cc83ea210"} Oct 30 16:51:12.171982 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.168219 4861 scope.go:122] "RemoveContainer" containerID="6b6d163e16d031dd34af88c7648301409573ea3b3d6482f8f6a04ecb34696f65" Oct 30 16:51:12.171982 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.168367 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz" Oct 30 16:51:12.177545 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.177511 4861 generic.go:356] "Generic (PLEG): container finished" podID="1916010e-81c5-4927-af79-d4d9b343ace4" containerID="bb8ac86aa8771f758ed4b7b8ed8b98045642f933fae2c33bd34eca5114822b17" exitCode=0 Oct 30 16:51:12.177713 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.177679 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" Oct 30 16:51:12.181210 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.177678 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" event={"ID":"1916010e-81c5-4927-af79-d4d9b343ace4","Type":"ContainerDied","Data":"bb8ac86aa8771f758ed4b7b8ed8b98045642f933fae2c33bd34eca5114822b17"} Oct 30 16:51:12.181210 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.177787 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz" event={"ID":"1916010e-81c5-4927-af79-d4d9b343ace4","Type":"ContainerDied","Data":"1c0c368bb39f0044f427f6207790e17f652f070f1151eae3fdee36607c290f68"} Oct 30 16:51:12.181210 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.180550 4861 generic.go:356] "Generic (PLEG): container finished" podID="5fa04564-6eb3-4184-b9e2-3d64ac6752ad" containerID="759c27d3db6949410211857f0c4ef5b3f1a7ee69b1fcb5f4a30f1b89d02f7958" exitCode=0 Oct 30 16:51:12.181582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.181559 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" Oct 30 16:51:12.186204 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.185694 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" event={"ID":"5fa04564-6eb3-4184-b9e2-3d64ac6752ad","Type":"ContainerDied","Data":"759c27d3db6949410211857f0c4ef5b3f1a7ee69b1fcb5f4a30f1b89d02f7958"} Oct 30 16:51:12.186204 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.185767 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl" event={"ID":"5fa04564-6eb3-4184-b9e2-3d64ac6752ad","Type":"ContainerDied","Data":"3bb473b27309ac6a664541ccdebdbf9d98827a15314c98a0c9cb2b7f6f626437"} Oct 30 16:51:12.194343 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.194316 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1916010e-81c5-4927-af79-d4d9b343ace4-kube-api-access-7phvm" pod "1916010e-81c5-4927-af79-d4d9b343ace4" (UID: "1916010e-81c5-4927-af79-d4d9b343ace4"). InnerVolumeSpecName "kube-api-access-7phvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:12.202095 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.202064 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e15e44-d162-4dc6-9c52-8ac1be42411f-kube-api-access-s7rr6" pod "d3e15e44-d162-4dc6-9c52-8ac1be42411f" (UID: "d3e15e44-d162-4dc6-9c52-8ac1be42411f"). InnerVolumeSpecName "kube-api-access-s7rr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:12.211775 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.211753 4861 scope.go:122] "RemoveContainer" containerID="6b6d163e16d031dd34af88c7648301409573ea3b3d6482f8f6a04ecb34696f65" Oct 30 16:51:12.212177 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:51:12.212151 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6d163e16d031dd34af88c7648301409573ea3b3d6482f8f6a04ecb34696f65\": container with ID starting with 6b6d163e16d031dd34af88c7648301409573ea3b3d6482f8f6a04ecb34696f65 not found: ID does not exist" containerID="6b6d163e16d031dd34af88c7648301409573ea3b3d6482f8f6a04ecb34696f65" Oct 30 16:51:12.212268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.212203 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6d163e16d031dd34af88c7648301409573ea3b3d6482f8f6a04ecb34696f65"} err="failed to get container status \"6b6d163e16d031dd34af88c7648301409573ea3b3d6482f8f6a04ecb34696f65\": rpc error: code = NotFound desc = could not find container \"6b6d163e16d031dd34af88c7648301409573ea3b3d6482f8f6a04ecb34696f65\": container with ID starting with 6b6d163e16d031dd34af88c7648301409573ea3b3d6482f8f6a04ecb34696f65 not found: ID does not exist" Oct 30 16:51:12.212268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.212230 4861 scope.go:122] "RemoveContainer" containerID="bb8ac86aa8771f758ed4b7b8ed8b98045642f933fae2c33bd34eca5114822b17" Oct 30 16:51:12.221548 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.221523 4861 scope.go:122] "RemoveContainer" containerID="bb8ac86aa8771f758ed4b7b8ed8b98045642f933fae2c33bd34eca5114822b17" Oct 30 16:51:12.221893 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:51:12.221857 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8ac86aa8771f758ed4b7b8ed8b98045642f933fae2c33bd34eca5114822b17\": container with ID starting with bb8ac86aa8771f758ed4b7b8ed8b98045642f933fae2c33bd34eca5114822b17 not found: ID does not exist" containerID="bb8ac86aa8771f758ed4b7b8ed8b98045642f933fae2c33bd34eca5114822b17" Oct 30 16:51:12.221988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.221898 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8ac86aa8771f758ed4b7b8ed8b98045642f933fae2c33bd34eca5114822b17"} err="failed to get container status \"bb8ac86aa8771f758ed4b7b8ed8b98045642f933fae2c33bd34eca5114822b17\": rpc error: code = NotFound desc = could not find container \"bb8ac86aa8771f758ed4b7b8ed8b98045642f933fae2c33bd34eca5114822b17\": container with ID starting with bb8ac86aa8771f758ed4b7b8ed8b98045642f933fae2c33bd34eca5114822b17 not found: ID does not exist" Oct 30 16:51:12.221988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.221916 4861 scope.go:122] "RemoveContainer" containerID="759c27d3db6949410211857f0c4ef5b3f1a7ee69b1fcb5f4a30f1b89d02f7958" Oct 30 16:51:12.231596 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.231577 4861 scope.go:122] "RemoveContainer" containerID="759c27d3db6949410211857f0c4ef5b3f1a7ee69b1fcb5f4a30f1b89d02f7958" Oct 30 16:51:12.231904 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:51:12.231881 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"759c27d3db6949410211857f0c4ef5b3f1a7ee69b1fcb5f4a30f1b89d02f7958\": container with ID starting with 759c27d3db6949410211857f0c4ef5b3f1a7ee69b1fcb5f4a30f1b89d02f7958 not found: ID does not exist" containerID="759c27d3db6949410211857f0c4ef5b3f1a7ee69b1fcb5f4a30f1b89d02f7958" Oct 30 16:51:12.232000 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.231908 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759c27d3db6949410211857f0c4ef5b3f1a7ee69b1fcb5f4a30f1b89d02f7958"} err="failed to get container status \"759c27d3db6949410211857f0c4ef5b3f1a7ee69b1fcb5f4a30f1b89d02f7958\": rpc error: code = NotFound desc = could not find container \"759c27d3db6949410211857f0c4ef5b3f1a7ee69b1fcb5f4a30f1b89d02f7958\": container with ID starting with 759c27d3db6949410211857f0c4ef5b3f1a7ee69b1fcb5f4a30f1b89d02f7958 not found: ID does not exist" Oct 30 16:51:12.243864 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.243838 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7rr6\" (UniqueName: \"kubernetes.io/projected/d3e15e44-d162-4dc6-9c52-8ac1be42411f-kube-api-access-s7rr6\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:12.243986 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.243869 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1916010e-81c5-4927-af79-d4d9b343ace4-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:12.243986 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.243883 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7phvm\" (UniqueName: \"kubernetes.io/projected/1916010e-81c5-4927-af79-d4d9b343ace4-kube-api-access-7phvm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:12.243986 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.243898 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e15e44-d162-4dc6-9c52-8ac1be42411f-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:12.255737 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.255697 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl"] Oct 30 16:51:12.260109 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.260087 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-06ac18-78b8b6b78-6vttl"] Oct 30 16:51:12.520113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.520072 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz"] Oct 30 16:51:12.524046 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.524016 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-06ac18-754bc5dc59-cn4tz"] Oct 30 16:51:12.564042 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.564011 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz"] Oct 30 16:51:12.567937 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:12.567907 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-06ac18-5bc6968db6-pz2qz"] Oct 30 16:51:13.807128 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:13.807088 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1916010e-81c5-4927-af79-d4d9b343ace4" path="/var/lib/kubelet/pods/1916010e-81c5-4927-af79-d4d9b343ace4/volumes" Oct 30 16:51:13.807658 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:13.807634 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa04564-6eb3-4184-b9e2-3d64ac6752ad" path="/var/lib/kubelet/pods/5fa04564-6eb3-4184-b9e2-3d64ac6752ad/volumes" Oct 30 16:51:13.808033 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:13.808015 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e15e44-d162-4dc6-9c52-8ac1be42411f" path="/var/lib/kubelet/pods/d3e15e44-d162-4dc6-9c52-8ac1be42411f/volumes" Oct 30 16:51:15.976302 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976265 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85"] Oct 30 16:51:15.976795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976780 4861 cpu_manager.go:414] "containerMap: removing container" podUID="1e318052-ec4f-406a-97ff-a1a99b698595" containerName="payload-processing" Oct 30 16:51:15.976843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976798 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e318052-ec4f-406a-97ff-a1a99b698595" containerName="payload-processing" Oct 30 16:51:15.976843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976811 4861 cpu_manager.go:414] "containerMap: removing container" podUID="dac9dbe1-f2bc-4feb-b107-b94cca178634" containerName="maas-api" Oct 30 16:51:15.976843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976816 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac9dbe1-f2bc-4feb-b107-b94cca178634" containerName="maas-api" Oct 30 16:51:15.976843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976828 4861 cpu_manager.go:414] "containerMap: removing container" podUID="5fa04564-6eb3-4184-b9e2-3d64ac6752ad" containerName="maas-api" Oct 30 16:51:15.976843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976834 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa04564-6eb3-4184-b9e2-3d64ac6752ad" containerName="maas-api" Oct 30 16:51:15.977010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976848 4861 cpu_manager.go:414] "containerMap: removing container" podUID="d3e15e44-d162-4dc6-9c52-8ac1be42411f" containerName="payload-pre-processing" Oct 30 16:51:15.977010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976852 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e15e44-d162-4dc6-9c52-8ac1be42411f" containerName="payload-pre-processing" Oct 30 16:51:15.977010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976872 4861 cpu_manager.go:414] "containerMap: removing container" podUID="63d93220-40d7-4b64-ba7b-9764ad028511" containerName="payload-pre-processing" Oct 30 16:51:15.977010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976877 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d93220-40d7-4b64-ba7b-9764ad028511" containerName="payload-pre-processing" Oct 30 16:51:15.977010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976886 4861 cpu_manager.go:414] "containerMap: removing container" podUID="1916010e-81c5-4927-af79-d4d9b343ace4" containerName="payload-processing" Oct 30 16:51:15.977010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976891 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1916010e-81c5-4927-af79-d4d9b343ace4" containerName="payload-processing" Oct 30 16:51:15.977010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976899 4861 cpu_manager.go:414] "containerMap: removing container" podUID="9db4fb72-7b88-4237-8c83-38f14f803caa" containerName="istio-proxy" Oct 30 16:51:15.977010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976904 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db4fb72-7b88-4237-8c83-38f14f803caa" containerName="istio-proxy" Oct 30 16:51:15.977010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976970 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e15e44-d162-4dc6-9c52-8ac1be42411f" containerName="payload-pre-processing" Oct 30 16:51:15.977010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976983 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c4a663-826a-4d68-9208-e16cb76839b5" containerName="istio-proxy" Oct 30 16:51:15.977010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976990 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d93220-40d7-4b64-ba7b-9764ad028511" containerName="payload-pre-processing" Oct 30 16:51:15.977010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.976996 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="1916010e-81c5-4927-af79-d4d9b343ace4" containerName="payload-processing" Oct 30 16:51:15.977010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.977003 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e318052-ec4f-406a-97ff-a1a99b698595" containerName="payload-processing" Oct 30 16:51:15.977010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.977007 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac9dbe1-f2bc-4feb-b107-b94cca178634" containerName="maas-api" Oct 30 16:51:15.977010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.977013 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db4fb72-7b88-4237-8c83-38f14f803caa" containerName="istio-proxy" Oct 30 16:51:15.977456 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.977018 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa04564-6eb3-4184-b9e2-3d64ac6752ad" containerName="maas-api" Oct 30 16:51:15.982109 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.982089 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:15.992651 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.992627 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh"] Oct 30 16:51:15.996906 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:15.996887 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.006296 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.006274 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85"] Oct 30 16:51:16.036255 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.036227 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh"] Oct 30 16:51:16.090142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090110 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.090309 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090146 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.090309 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090166 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.090396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090314 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.090396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090371 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.090490 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090396 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.090490 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090426 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.090490 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090472 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.090696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090572 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.090696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090595 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.090696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090635 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.090922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090692 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.090922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090787 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27kh4\" (UniqueName: \"kubernetes.io/projected/879aadbb-7e84-4ffc-b078-925a5ff30c43-kube-api-access-27kh4\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.090922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090819 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.090922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090850 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/879aadbb-7e84-4ffc-b078-925a5ff30c43-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.090922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090918 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.091123 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090943 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.091123 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.090962 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd24w\" (UniqueName: \"kubernetes.io/projected/e12f55db-ef70-4f34-b3f7-14d431f1ca78-kube-api-access-vd24w\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.191633 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.191605 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.191789 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.191640 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.191789 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.191662 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd24w\" (UniqueName: \"kubernetes.io/projected/e12f55db-ef70-4f34-b3f7-14d431f1ca78-kube-api-access-vd24w\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.191789 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.191704 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.191789 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.191764 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.192027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.191792 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.192027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.191842 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.192027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.191894 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.192027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.191930 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.192027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.191964 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.192282 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.192113 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.192282 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.192192 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.192282 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.192204 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.192282 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.192245 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.192282 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.192266 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.192282 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.192280 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.192577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.192305 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.192577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.192338 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.192577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.192338 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.192577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.192392 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.192577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.192451 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27kh4\" (UniqueName: \"kubernetes.io/projected/879aadbb-7e84-4ffc-b078-925a5ff30c43-kube-api-access-27kh4\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.192577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.192482 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.192577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.192511 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.192907 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.192593 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.192907 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.192515 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/879aadbb-7e84-4ffc-b078-925a5ff30c43-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.193025 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.192931 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.193101 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.193072 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.193257 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.193137 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/879aadbb-7e84-4ffc-b078-925a5ff30c43-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.194880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.194851 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.195122 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.195101 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.195238 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.195144 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.195443 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.195421 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.216823 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.216802 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd24w\" (UniqueName: \"kubernetes.io/projected/e12f55db-ef70-4f34-b3f7-14d431f1ca78-kube-api-access-vd24w\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.218870 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.218839 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.219010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.218985 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.219054 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.219026 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27kh4\" (UniqueName: \"kubernetes.io/projected/879aadbb-7e84-4ffc-b078-925a5ff30c43-kube-api-access-27kh4\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.296628 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.296570 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:16.314001 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.313983 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:16.649845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.649786 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85"] Oct 30 16:51:16.652247 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:51:16.652224 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879aadbb_7e84_4ffc_b078_925a5ff30c43.slice/crio-060b8a46c456e61cba17721627ac7c4cf88b217d9a9998d82885181e69acddb7 WatchSource:0}: Error finding container 060b8a46c456e61cba17721627ac7c4cf88b217d9a9998d82885181e69acddb7: Status 404 returned error can't find the container with id 060b8a46c456e61cba17721627ac7c4cf88b217d9a9998d82885181e69acddb7 Oct 30 16:51:16.655024 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:51:16.654980 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12f55db_ef70_4f34_b3f7_14d431f1ca78.slice/crio-5fdee313cc7a8855a172e80e57fcc9bb551a318c92467e6f0ba8a19c307bc50b WatchSource:0}: Error finding container 5fdee313cc7a8855a172e80e57fcc9bb551a318c92467e6f0ba8a19c307bc50b: Status 404 returned error can't find the container with id 5fdee313cc7a8855a172e80e57fcc9bb551a318c92467e6f0ba8a19c307bc50b Oct 30 16:51:16.655235 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.655203 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:16.655312 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.655270 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:16.655312 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.655278 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh"] Oct 30 16:51:16.655312 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.655301 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:16.657379 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.657352 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:16.657464 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.657410 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:16.657464 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:16.657441 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:17.217735 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:17.217685 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" event={"ID":"e12f55db-ef70-4f34-b3f7-14d431f1ca78","Type":"ContainerStarted","Data":"a21cbba253f2b833af7f81fb151c5a62dc67d5d6fb737d89c14a2300c40b5531"} Oct 30 16:51:17.218200 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:17.217781 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" event={"ID":"e12f55db-ef70-4f34-b3f7-14d431f1ca78","Type":"ContainerStarted","Data":"5fdee313cc7a8855a172e80e57fcc9bb551a318c92467e6f0ba8a19c307bc50b"} Oct 30 16:51:17.219655 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:17.219632 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" event={"ID":"879aadbb-7e84-4ffc-b078-925a5ff30c43","Type":"ContainerStarted","Data":"0f6d4f29ba76a33adc3b7c182e44c28072a436f271dc351a8cb18afbc6dd54f5"} Oct 30 16:51:17.219815 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:17.219675 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" event={"ID":"879aadbb-7e84-4ffc-b078-925a5ff30c43","Type":"ContainerStarted","Data":"060b8a46c456e61cba17721627ac7c4cf88b217d9a9998d82885181e69acddb7"} Oct 30 16:51:17.265066 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:17.264998 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" podStartSLOduration=2.264986296 podStartE2EDuration="2.264986296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:17.264837254 +0000 UTC m=+1707.607511292" watchObservedRunningTime="2026-10-30 16:51:17.264986296 +0000 UTC m=+1707.607660333" Oct 30 16:51:17.297686 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:17.297662 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:17.298936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:17.298916 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:51:17.298987 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:17.298958 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" podUID="879aadbb-7e84-4ffc-b078-925a5ff30c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 16:51:17.314177 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:17.314154 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:17.315814 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:17.315783 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 16:51:17.315927 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:17.315836 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" podUID="e12f55db-ef70-4f34-b3f7-14d431f1ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 16:51:17.317107 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:17.317066 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" podStartSLOduration=2.317054781 podStartE2EDuration="2.317054781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:17.316189345 +0000 UTC m=+1707.658863383" watchObservedRunningTime="2026-10-30 16:51:17.317054781 +0000 UTC m=+1707.659728818" Oct 30 16:51:18.297340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:18.297307 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:51:18.297770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:18.297373 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" podUID="879aadbb-7e84-4ffc-b078-925a5ff30c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 16:51:18.320750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:18.316188 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 16:51:18.320750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:18.316247 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" podUID="e12f55db-ef70-4f34-b3f7-14d431f1ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 16:51:19.297514 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:19.297485 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:51:19.297992 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:19.297868 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" podUID="879aadbb-7e84-4ffc-b078-925a5ff30c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 16:51:19.328744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:19.328620 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 16:51:19.328744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:19.328675 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" podUID="e12f55db-ef70-4f34-b3f7-14d431f1ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 16:51:20.289539 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:51:20.289497 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:51:20.297179 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:20.297062 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:51:20.297179 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:20.297138 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" podUID="879aadbb-7e84-4ffc-b078-925a5ff30c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 16:51:20.330751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:20.320824 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 16:51:20.330751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:20.320883 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" podUID="e12f55db-ef70-4f34-b3f7-14d431f1ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 16:51:21.297531 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:21.297497 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:51:21.297805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:21.297556 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" podUID="879aadbb-7e84-4ffc-b078-925a5ff30c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 16:51:21.329755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:21.329567 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 16:51:21.329755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:21.329621 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" podUID="e12f55db-ef70-4f34-b3f7-14d431f1ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 16:51:22.297099 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.297058 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:51:22.297488 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.297121 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" podUID="879aadbb-7e84-4ffc-b078-925a5ff30c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 16:51:22.315113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.315078 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 16:51:22.316667 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.316636 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" podUID="e12f55db-ef70-4f34-b3f7-14d431f1ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 16:51:22.712190 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.711448 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m"] Oct 30 16:51:22.764745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.756910 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:22.785098 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.783101 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-52f3fd-openshift-default-dockercfg-4drzc\"" Oct 30 16:51:22.796747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.796453 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m"] Oct 30 16:51:22.903764 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.889469 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwpxg\" (UniqueName: \"kubernetes.io/projected/963c434d-5cb8-4c34-bc1c-388e3cb1e125-kube-api-access-zwpxg\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:22.989555 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.952979 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-token\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:22.989555 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.953050 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-workload-certs\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:22.989555 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.953143 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-credential-socket\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:22.989555 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.953202 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-envoy\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:22.989555 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.953245 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-podinfo\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:22.989555 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.953285 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-workload-socket\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:22.989555 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.953319 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-data\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:22.989555 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:22.953368 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istiod-ca-cert\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.065751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.065421 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwpxg\" (UniqueName: \"kubernetes.io/projected/963c434d-5cb8-4c34-bc1c-388e3cb1e125-kube-api-access-zwpxg\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.065751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.065521 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-token\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.065751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.065580 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-workload-certs\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.065751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.065650 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-credential-socket\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.065751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.065699 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-envoy\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.065751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.065754 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-podinfo\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.066234 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.065789 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-workload-socket\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.066234 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.065822 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-data\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.066234 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.065866 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istiod-ca-cert\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.078748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.067023 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-credential-socket\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.078748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.067830 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-workload-certs\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.078748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.068230 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istiod-ca-cert\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.078748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.068475 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-workload-socket\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.078748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.068701 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-data\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.083399 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.083358 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-envoy\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.117913 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.117842 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-token\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.119419 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.119380 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-podinfo\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.131764 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.128917 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwpxg\" (UniqueName: \"kubernetes.io/projected/963c434d-5cb8-4c34-bc1c-388e3cb1e125-kube-api-access-zwpxg\") pod \"e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.171254 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.170967 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:23.297627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.297086 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:51:23.297627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.297139 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" podUID="879aadbb-7e84-4ffc-b078-925a5ff30c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 16:51:23.323752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.320156 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 16:51:23.323752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:23.320209 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" podUID="e12f55db-ef70-4f34-b3f7-14d431f1ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 16:51:24.228248 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:24.228215 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m"] Oct 30 16:51:24.351798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:24.348737 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:51:24.351798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:24.348814 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" podUID="879aadbb-7e84-4ffc-b078-925a5ff30c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 16:51:24.352911 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:24.352881 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 16:51:24.353156 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:24.353105 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" podUID="e12f55db-ef70-4f34-b3f7-14d431f1ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 16:51:24.388971 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:24.388927 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:24.389286 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:24.389262 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:24.389518 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:24.389492 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:24.392187 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:24.392164 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" event={"ID":"963c434d-5cb8-4c34-bc1c-388e3cb1e125","Type":"ContainerStarted","Data":"195dbd493756818d90f7418095627fac9b3199dbac58dff51e488a149378ad79"} Oct 30 16:51:25.297517 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:25.297477 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:51:25.297708 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:25.297541 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" podUID="879aadbb-7e84-4ffc-b078-925a5ff30c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Oct 30 16:51:25.315200 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:25.315166 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Oct 30 16:51:25.315332 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:25.315225 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" podUID="e12f55db-ef70-4f34-b3f7-14d431f1ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Oct 30 16:51:25.402826 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:25.402772 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" event={"ID":"963c434d-5cb8-4c34-bc1c-388e3cb1e125","Type":"ContainerStarted","Data":"1ec12ff8a0117350f2169865ef2a67e7d6d587e6fe51e906c512605d7cb5c01d"} Oct 30 16:51:25.467785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:25.467715 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" podStartSLOduration=3.467702263 podStartE2EDuration="3.467702263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:25.46594584 +0000 UTC m=+1715.808619890" watchObservedRunningTime="2026-10-30 16:51:25.467702263 +0000 UTC m=+1715.810376301" Oct 30 16:51:26.173070 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:26.173028 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:26.179139 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:26.179109 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:26.409168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:26.409129 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:26.410076 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:26.410059 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:51:27.339889 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:27.339850 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:51:27.341002 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:27.340964 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" podUID="879aadbb-7e84-4ffc-b078-925a5ff30c43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:51:27.358685 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:27.358287 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:51:27.358685 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:27.358344 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" podUID="e12f55db-ef70-4f34-b3f7-14d431f1ca78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:51:27.636504 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:27.636418 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:27.636917 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:27.636897 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:27.638140 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:27.638117 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:51:27.673945 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:27.673917 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:27.674443 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:27.674417 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:27.675349 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:27.675328 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:51:29.972856 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:29.972798 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s"] Oct 30 16:51:30.065792 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.058211 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s"] Oct 30 16:51:30.066101 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.058699 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.185990 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.185953 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.186148 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.185992 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.186148 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.186073 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.186244 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.186194 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxh6r\" (UniqueName: \"kubernetes.io/projected/16535ec9-31d5-4620-b9c9-2367ba14e26d-kube-api-access-hxh6r\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.186293 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.186255 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.186333 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.186305 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.186381 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.186335 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16535ec9-31d5-4620-b9c9-2367ba14e26d-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.186381 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.186355 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.186487 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.186462 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.287420 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.287351 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxh6r\" (UniqueName: \"kubernetes.io/projected/16535ec9-31d5-4620-b9c9-2367ba14e26d-kube-api-access-hxh6r\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.287420 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.287408 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.287648 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.287433 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.287648 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.287449 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16535ec9-31d5-4620-b9c9-2367ba14e26d-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.287648 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.287465 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.287648 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.287484 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.287648 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.287555 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.287974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.287705 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.287974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.287791 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.287974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.287930 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.288134 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.288092 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.288187 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.288156 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.288288 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.288260 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.288407 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.288387 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16535ec9-31d5-4620-b9c9-2367ba14e26d-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.290087 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.290062 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.290294 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.290274 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.311263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.311243 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.311479 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.311458 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxh6r\" (UniqueName: \"kubernetes.io/projected/16535ec9-31d5-4620-b9c9-2367ba14e26d-kube-api-access-hxh6r\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.392341 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.392305 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:30.715835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.715804 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s"] Oct 30 16:51:30.716967 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:51:30.716941 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16535ec9_31d5_4620_b9c9_2367ba14e26d.slice/crio-954fc773ef988a28aece4b1c24d2c0677128e4fb2eed70d71535fad47a40f079 WatchSource:0}: Error finding container 954fc773ef988a28aece4b1c24d2c0677128e4fb2eed70d71535fad47a40f079: Status 404 returned error can't find the container with id 954fc773ef988a28aece4b1c24d2c0677128e4fb2eed70d71535fad47a40f079 Oct 30 16:51:30.719436 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.719404 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:30.719500 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.719472 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:30.719535 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:30.719501 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:31.448599 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:31.448557 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" event={"ID":"16535ec9-31d5-4620-b9c9-2367ba14e26d","Type":"ContainerStarted","Data":"15893985a97ed68896f6e550193012a9e14d1f9824477bfa644f09595a11642f"} Oct 30 16:51:31.449077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:31.448620 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" event={"ID":"16535ec9-31d5-4620-b9c9-2367ba14e26d","Type":"ContainerStarted","Data":"954fc773ef988a28aece4b1c24d2c0677128e4fb2eed70d71535fad47a40f079"} Oct 30 16:51:31.516745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:31.509311 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" podStartSLOduration=2.509291462 podStartE2EDuration="2.509291462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:31.508135356 +0000 UTC m=+1721.850809395" watchObservedRunningTime="2026-10-30 16:51:31.509291462 +0000 UTC m=+1721.851965501" Oct 30 16:51:32.392572 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:32.392533 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:32.394564 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:32.394535 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 16:51:32.394698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:32.394586 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" podUID="16535ec9-31d5-4620-b9c9-2367ba14e26d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 16:51:33.393057 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:33.393020 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 16:51:33.393454 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:33.393076 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" podUID="16535ec9-31d5-4620-b9c9-2367ba14e26d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 16:51:34.393670 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:34.393632 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 16:51:34.394087 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:34.393685 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" podUID="16535ec9-31d5-4620-b9c9-2367ba14e26d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 16:51:35.392937 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:35.392903 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 16:51:35.393117 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:35.392959 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" podUID="16535ec9-31d5-4620-b9c9-2367ba14e26d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 16:51:36.393743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:36.393687 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 16:51:36.394143 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:36.393769 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" podUID="16535ec9-31d5-4620-b9c9-2367ba14e26d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 16:51:37.393385 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:37.393347 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 16:51:37.393544 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:37.393402 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" podUID="16535ec9-31d5-4620-b9c9-2367ba14e26d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 16:51:38.397537 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:38.397507 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:38.398007 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:38.397942 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:38.398647 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:38.398626 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:51:45.025514 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.025477 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh"] Oct 30 16:51:45.055044 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.055009 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.216748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.209906 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.216748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.209976 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.216748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.210012 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.216748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.210057 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.216748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.210109 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.216748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.210156 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9bf5\" (UniqueName: \"kubernetes.io/projected/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-kube-api-access-x9bf5\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.216748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.210199 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.216748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.210260 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.216748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.210324 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.400817 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.355543 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh"] Oct 30 16:51:45.426749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.371501 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.476746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.468914 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.476746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.469026 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.476746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.469091 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.476746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.469121 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.476746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.469173 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.476746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.469235 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.519748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.481767 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.520062 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.520002 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.520782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.520757 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9bf5\" (UniqueName: \"kubernetes.io/projected/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-kube-api-access-x9bf5\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.543845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.543819 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.545192 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.545152 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.545497 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.545467 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.546471 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.546435 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.583072 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.583042 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9bf5\" (UniqueName: \"kubernetes.io/projected/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-kube-api-access-x9bf5\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.597744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.595228 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.630345 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.620826 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.630345 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.625674 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:45.787754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:45.774928 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:46.383384 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:46.383346 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n"] Oct 30 16:51:46.396229 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:46.396196 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr"] Oct 30 16:51:46.396819 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:46.396782 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" podUID="e80e615e-5678-4264-a1ac-7bd053f88f3f" containerName="payload-processing" containerID="cri-o://e4aff3d5879fe7d87ae2b0924cfa8df063adef9f402f9dacd8861acd0c3cbb00" gracePeriod=30 Oct 30 16:51:46.402758 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:46.400390 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" podUID="d61c6199-58ad-4ff1-b662-0fa4f2d9576a" containerName="payload-pre-processing" containerID="cri-o://670e456513bf4c0a59da12777eeca52e1de4ac6f7b5fe32818c0691200104472" gracePeriod=30 Oct 30 16:51:46.846587 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:46.846550 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh"] Oct 30 16:51:46.853147 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:51:46.853113 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a6ea1d_68b5_4225_a5ac_d6e8fc3b19c8.slice/crio-88b41789fdab356b487e452c3c14f9879bcbe1f5823453e3ccf1caceaf8e46ba WatchSource:0}: Error finding container 88b41789fdab356b487e452c3c14f9879bcbe1f5823453e3ccf1caceaf8e46ba: Status 404 returned error can't find the container with id 88b41789fdab356b487e452c3c14f9879bcbe1f5823453e3ccf1caceaf8e46ba Oct 30 16:51:46.860100 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:46.856497 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:46.860100 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:46.856590 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:46.860100 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:46.856637 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:51:46.894894 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:46.894871 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" Oct 30 16:51:46.929399 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:46.926396 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" Oct 30 16:51:47.002599 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.002570 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e80e615e-5678-4264-a1ac-7bd053f88f3f-kube-api-access-5sh4w\" (UniqueName: \"kubernetes.io/projected/e80e615e-5678-4264-a1ac-7bd053f88f3f-kube-api-access-5sh4w\") pod \"e80e615e-5678-4264-a1ac-7bd053f88f3f\" (UID: \"e80e615e-5678-4264-a1ac-7bd053f88f3f\") " Oct 30 16:51:47.002821 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.002623 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61c6199-58ad-4ff1-b662-0fa4f2d9576a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61c6199-58ad-4ff1-b662-0fa4f2d9576a-plugins-config-volume\") pod \"d61c6199-58ad-4ff1-b662-0fa4f2d9576a\" (UID: \"d61c6199-58ad-4ff1-b662-0fa4f2d9576a\") " Oct 30 16:51:47.002821 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.002737 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61c6199-58ad-4ff1-b662-0fa4f2d9576a-kube-api-access-z9bvr\" (UniqueName: \"kubernetes.io/projected/d61c6199-58ad-4ff1-b662-0fa4f2d9576a-kube-api-access-z9bvr\") pod \"d61c6199-58ad-4ff1-b662-0fa4f2d9576a\" (UID: \"d61c6199-58ad-4ff1-b662-0fa4f2d9576a\") " Oct 30 16:51:47.003700 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.003664 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61c6199-58ad-4ff1-b662-0fa4f2d9576a-plugins-config-volume" pod "d61c6199-58ad-4ff1-b662-0fa4f2d9576a" (UID: "d61c6199-58ad-4ff1-b662-0fa4f2d9576a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:47.003871 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.003736 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e80e615e-5678-4264-a1ac-7bd053f88f3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80e615e-5678-4264-a1ac-7bd053f88f3f-plugins-config-volume\") pod \"e80e615e-5678-4264-a1ac-7bd053f88f3f\" (UID: \"e80e615e-5678-4264-a1ac-7bd053f88f3f\") " Oct 30 16:51:47.003871 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.003758 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80e615e-5678-4264-a1ac-7bd053f88f3f-plugins-config-volume" pod "e80e615e-5678-4264-a1ac-7bd053f88f3f" (UID: "e80e615e-5678-4264-a1ac-7bd053f88f3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:47.004261 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.004243 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e80e615e-5678-4264-a1ac-7bd053f88f3f-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:47.004330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.004267 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d61c6199-58ad-4ff1-b662-0fa4f2d9576a-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:47.006342 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.006317 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61c6199-58ad-4ff1-b662-0fa4f2d9576a-kube-api-access-z9bvr" pod "d61c6199-58ad-4ff1-b662-0fa4f2d9576a" (UID: "d61c6199-58ad-4ff1-b662-0fa4f2d9576a"). InnerVolumeSpecName "kube-api-access-z9bvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:47.006603 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.006576 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80e615e-5678-4264-a1ac-7bd053f88f3f-kube-api-access-5sh4w" pod "e80e615e-5678-4264-a1ac-7bd053f88f3f" (UID: "e80e615e-5678-4264-a1ac-7bd053f88f3f"). InnerVolumeSpecName "kube-api-access-5sh4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:47.105843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.105809 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sh4w\" (UniqueName: \"kubernetes.io/projected/e80e615e-5678-4264-a1ac-7bd053f88f3f-kube-api-access-5sh4w\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:47.105843 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.105833 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9bvr\" (UniqueName: \"kubernetes.io/projected/d61c6199-58ad-4ff1-b662-0fa4f2d9576a-kube-api-access-z9bvr\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:51:47.632825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.632784 4861 generic.go:356] "Generic (PLEG): container finished" podID="e80e615e-5678-4264-a1ac-7bd053f88f3f" containerID="e4aff3d5879fe7d87ae2b0924cfa8df063adef9f402f9dacd8861acd0c3cbb00" exitCode=0 Oct 30 16:51:47.633323 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.632883 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" Oct 30 16:51:47.633323 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.632900 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" event={"ID":"e80e615e-5678-4264-a1ac-7bd053f88f3f","Type":"ContainerDied","Data":"e4aff3d5879fe7d87ae2b0924cfa8df063adef9f402f9dacd8861acd0c3cbb00"} Oct 30 16:51:47.633323 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.632969 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n" event={"ID":"e80e615e-5678-4264-a1ac-7bd053f88f3f","Type":"ContainerDied","Data":"9e5f8816c790d92961696dea407316ae27821a1b0daac3220d6715de68a82972"} Oct 30 16:51:47.633323 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.633000 4861 scope.go:122] "RemoveContainer" containerID="e4aff3d5879fe7d87ae2b0924cfa8df063adef9f402f9dacd8861acd0c3cbb00" Oct 30 16:51:47.635028 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.634996 4861 generic.go:356] "Generic (PLEG): container finished" podID="d61c6199-58ad-4ff1-b662-0fa4f2d9576a" containerID="670e456513bf4c0a59da12777eeca52e1de4ac6f7b5fe32818c0691200104472" exitCode=0 Oct 30 16:51:47.635248 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.635067 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" Oct 30 16:51:47.635366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.635084 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" event={"ID":"d61c6199-58ad-4ff1-b662-0fa4f2d9576a","Type":"ContainerDied","Data":"670e456513bf4c0a59da12777eeca52e1de4ac6f7b5fe32818c0691200104472"} Oct 30 16:51:47.635435 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.635379 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr" event={"ID":"d61c6199-58ad-4ff1-b662-0fa4f2d9576a","Type":"ContainerDied","Data":"8bc050d743c9a9b555044d0e7a6802a3a29ae5ef52f731c3f5d64f4ac062b75e"} Oct 30 16:51:47.640157 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.640127 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" event={"ID":"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8","Type":"ContainerStarted","Data":"119215ef6d621ae288697b0bda15e01342d36c9ff5ec0d4704f23f5e9a66d5a9"} Oct 30 16:51:47.640266 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.640170 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" event={"ID":"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8","Type":"ContainerStarted","Data":"88b41789fdab356b487e452c3c14f9879bcbe1f5823453e3ccf1caceaf8e46ba"} Oct 30 16:51:47.644578 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.644564 4861 scope.go:122] "RemoveContainer" containerID="e4aff3d5879fe7d87ae2b0924cfa8df063adef9f402f9dacd8861acd0c3cbb00" Oct 30 16:51:47.644896 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:51:47.644874 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4aff3d5879fe7d87ae2b0924cfa8df063adef9f402f9dacd8861acd0c3cbb00\": container with ID starting with e4aff3d5879fe7d87ae2b0924cfa8df063adef9f402f9dacd8861acd0c3cbb00 not found: ID does not exist" containerID="e4aff3d5879fe7d87ae2b0924cfa8df063adef9f402f9dacd8861acd0c3cbb00" Oct 30 16:51:47.644963 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.644906 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4aff3d5879fe7d87ae2b0924cfa8df063adef9f402f9dacd8861acd0c3cbb00"} err="failed to get container status \"e4aff3d5879fe7d87ae2b0924cfa8df063adef9f402f9dacd8861acd0c3cbb00\": rpc error: code = NotFound desc = could not find container \"e4aff3d5879fe7d87ae2b0924cfa8df063adef9f402f9dacd8861acd0c3cbb00\": container with ID starting with e4aff3d5879fe7d87ae2b0924cfa8df063adef9f402f9dacd8861acd0c3cbb00 not found: ID does not exist" Oct 30 16:51:47.644963 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.644921 4861 scope.go:122] "RemoveContainer" containerID="670e456513bf4c0a59da12777eeca52e1de4ac6f7b5fe32818c0691200104472" Oct 30 16:51:47.653785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.653761 4861 scope.go:122] "RemoveContainer" containerID="670e456513bf4c0a59da12777eeca52e1de4ac6f7b5fe32818c0691200104472" Oct 30 16:51:47.654067 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:51:47.654050 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670e456513bf4c0a59da12777eeca52e1de4ac6f7b5fe32818c0691200104472\": container with ID starting with 670e456513bf4c0a59da12777eeca52e1de4ac6f7b5fe32818c0691200104472 not found: ID does not exist" containerID="670e456513bf4c0a59da12777eeca52e1de4ac6f7b5fe32818c0691200104472" Oct 30 16:51:47.654169 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.654073 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670e456513bf4c0a59da12777eeca52e1de4ac6f7b5fe32818c0691200104472"} err="failed to get container status \"670e456513bf4c0a59da12777eeca52e1de4ac6f7b5fe32818c0691200104472\": rpc error: code = NotFound desc = could not find container \"670e456513bf4c0a59da12777eeca52e1de4ac6f7b5fe32818c0691200104472\": container with ID starting with 670e456513bf4c0a59da12777eeca52e1de4ac6f7b5fe32818c0691200104472 not found: ID does not exist" Oct 30 16:51:47.687154 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.687134 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr"] Oct 30 16:51:47.699509 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.699483 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6d5903-7bffd48867-s4qkr"] Oct 30 16:51:47.749239 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.749157 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n"] Oct 30 16:51:47.752714 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.752685 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6d5903-59b479959d-q7d2n"] Oct 30 16:51:47.784777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.784753 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:47.786404 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.786379 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 16:51:47.786507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.786421 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" podUID="84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 16:51:47.798777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.798701 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" podStartSLOduration=3.798684602 podStartE2EDuration="3.798684602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:47.796002948 +0000 UTC m=+1738.138676987" watchObservedRunningTime="2026-10-30 16:51:47.798684602 +0000 UTC m=+1738.141358639" Oct 30 16:51:47.832614 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.832580 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61c6199-58ad-4ff1-b662-0fa4f2d9576a" path="/var/lib/kubelet/pods/d61c6199-58ad-4ff1-b662-0fa4f2d9576a/volumes" Oct 30 16:51:47.833124 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:47.833107 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e80e615e-5678-4264-a1ac-7bd053f88f3f" path="/var/lib/kubelet/pods/e80e615e-5678-4264-a1ac-7bd053f88f3f/volumes" Oct 30 16:51:48.785077 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:48.785039 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 16:51:48.785560 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:48.785099 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" podUID="84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 16:51:49.784452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:49.784421 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 16:51:49.784678 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:49.784488 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" podUID="84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 16:51:50.290198 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:51:50.290161 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:51:50.784292 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:50.784261 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 16:51:50.784520 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:50.784497 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" podUID="84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 16:51:50.888975 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:50.888946 4861 scope.go:122] "RemoveContainer" containerID="1b8f9b0624d4be8c381d1cd208566d752a17f871245576d8c365aadfab04fe65" Oct 30 16:51:51.784950 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:51.784912 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 16:51:51.785438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:51.784970 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" podUID="84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 16:51:52.784966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:52.784930 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 16:51:52.785338 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:52.784983 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" podUID="84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 16:51:53.784353 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:53.784309 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 16:51:53.784552 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:53.784383 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" podUID="84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 16:51:54.836750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:54.834970 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:54.836750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:54.836406 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:51:54.849755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:51:54.848006 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:52:01.143459 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.143386 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg"] Oct 30 16:52:01.143921 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.143907 4861 cpu_manager.go:414] "containerMap: removing container" podUID="d61c6199-58ad-4ff1-b662-0fa4f2d9576a" containerName="payload-pre-processing" Oct 30 16:52:01.143966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.143922 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61c6199-58ad-4ff1-b662-0fa4f2d9576a" containerName="payload-pre-processing" Oct 30 16:52:01.144000 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.143967 4861 cpu_manager.go:414] "containerMap: removing container" podUID="e80e615e-5678-4264-a1ac-7bd053f88f3f" containerName="payload-processing" Oct 30 16:52:01.144000 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.143973 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e80e615e-5678-4264-a1ac-7bd053f88f3f" containerName="payload-processing" Oct 30 16:52:01.144068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.144055 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="e80e615e-5678-4264-a1ac-7bd053f88f3f" containerName="payload-processing" Oct 30 16:52:01.144068 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.144065 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61c6199-58ad-4ff1-b662-0fa4f2d9576a" containerName="payload-pre-processing" Oct 30 16:52:01.148827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.148807 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.175121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.175094 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg"] Oct 30 16:52:01.264115 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.264080 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.264115 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.264117 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.264363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.264145 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.264363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.264238 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.264363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.264265 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.264530 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.264408 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.264530 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.264496 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np5kx\" (UniqueName: \"kubernetes.io/projected/b22f4e7a-ed39-41e3-9983-253a897bf603-kube-api-access-np5kx\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.264634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.264530 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.264691 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.264635 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22f4e7a-ed39-41e3-9983-253a897bf603-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.365334 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.365304 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.365507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.365339 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.365507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.365415 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.365507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.365471 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np5kx\" (UniqueName: \"kubernetes.io/projected/b22f4e7a-ed39-41e3-9983-253a897bf603-kube-api-access-np5kx\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.365507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.365497 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.365790 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.365660 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22f4e7a-ed39-41e3-9983-253a897bf603-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.365863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.365790 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.365863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.365825 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.365863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.365850 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.366036 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.365860 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.366036 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.365929 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.366151 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.366077 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.366213 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.366179 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.366424 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.366395 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22f4e7a-ed39-41e3-9983-253a897bf603-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.368176 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.368152 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.368256 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.368246 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.389889 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.389862 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np5kx\" (UniqueName: \"kubernetes.io/projected/b22f4e7a-ed39-41e3-9983-253a897bf603-kube-api-access-np5kx\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.390779 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.390759 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.463407 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.463333 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:01.789876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.789849 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg"] Oct 30 16:52:01.790703 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:52:01.790679 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22f4e7a_ed39_41e3_9983_253a897bf603.slice/crio-e10d9eacded89726d9b813005d0e58945a03f015f036e642770668628c4cf328 WatchSource:0}: Error finding container e10d9eacded89726d9b813005d0e58945a03f015f036e642770668628c4cf328: Status 404 returned error can't find the container with id e10d9eacded89726d9b813005d0e58945a03f015f036e642770668628c4cf328 Oct 30 16:52:01.793105 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.793073 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:01.793196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.793141 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:01.793196 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:01.793170 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:02.156465 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:02.156376 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz"] Oct 30 16:52:02.161421 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:02.161400 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" Oct 30 16:52:02.178484 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:02.178455 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-07cb3c-dockercfg-hv5x7\"" Oct 30 16:52:02.179859 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:02.179837 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-07cb3c\"" Oct 30 16:52:02.180595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:02.180573 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz"] Oct 30 16:52:02.348145 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:02.348105 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033a9d36-1eef-48f4-8615-f96fa569eee5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz\" (UID: \"033a9d36-1eef-48f4-8615-f96fa569eee5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" Oct 30 16:52:02.348318 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:02.348200 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trmzx\" (UniqueName: \"kubernetes.io/projected/033a9d36-1eef-48f4-8615-f96fa569eee5-kube-api-access-trmzx\") pod \"payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz\" (UID: \"033a9d36-1eef-48f4-8615-f96fa569eee5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" Oct 30 16:52:02.466583 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:02.466503 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033a9d36-1eef-48f4-8615-f96fa569eee5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz\" (UID: \"033a9d36-1eef-48f4-8615-f96fa569eee5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" Oct 30 16:52:02.466583 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:02.466556 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trmzx\" (UniqueName: \"kubernetes.io/projected/033a9d36-1eef-48f4-8615-f96fa569eee5-kube-api-access-trmzx\") pod \"payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz\" (UID: \"033a9d36-1eef-48f4-8615-f96fa569eee5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" Oct 30 16:52:02.467410 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:02.467383 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033a9d36-1eef-48f4-8615-f96fa569eee5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz\" (UID: \"033a9d36-1eef-48f4-8615-f96fa569eee5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" Oct 30 16:52:02.490950 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:02.490925 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trmzx\" (UniqueName: \"kubernetes.io/projected/033a9d36-1eef-48f4-8615-f96fa569eee5-kube-api-access-trmzx\") pod \"payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz\" (UID: \"033a9d36-1eef-48f4-8615-f96fa569eee5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" Oct 30 16:52:02.764655 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:02.764616 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" event={"ID":"b22f4e7a-ed39-41e3-9983-253a897bf603","Type":"ContainerStarted","Data":"801cebc106511cc7c00bdaa0e4a261f6a20ae059f00efa40b7f083ea2a34eb50"} Oct 30 16:52:02.764884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:02.764683 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" event={"ID":"b22f4e7a-ed39-41e3-9983-253a897bf603","Type":"ContainerStarted","Data":"e10d9eacded89726d9b813005d0e58945a03f015f036e642770668628c4cf328"} Oct 30 16:52:02.783659 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:02.783631 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" Oct 30 16:52:02.858738 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:02.858661 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" podStartSLOduration=1.8586408639999998 podStartE2EDuration="1.858640864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:02.838706635 +0000 UTC m=+1753.181380672" watchObservedRunningTime="2026-10-30 16:52:02.858640864 +0000 UTC m=+1753.201314902" Oct 30 16:52:03.138797 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:03.138217 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz"] Oct 30 16:52:03.141161 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:52:03.141129 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033a9d36_1eef_48f4_8615_f96fa569eee5.slice/crio-feb77340abe005af250cd298abdfdda6494110da6b7fcc5742827a3b19f27f3c WatchSource:0}: Error finding container feb77340abe005af250cd298abdfdda6494110da6b7fcc5742827a3b19f27f3c: Status 404 returned error can't find the container with id feb77340abe005af250cd298abdfdda6494110da6b7fcc5742827a3b19f27f3c Oct 30 16:52:03.463827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:03.463709 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:03.465389 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:03.465363 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 16:52:03.465517 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:03.465413 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" podUID="b22f4e7a-ed39-41e3-9983-253a897bf603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 16:52:03.773191 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:03.773158 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" event={"ID":"033a9d36-1eef-48f4-8615-f96fa569eee5","Type":"ContainerStarted","Data":"86e21b3c15d997e93a7d54ace82e56f66f9c4a80ecea7bd57c620fca78f6a3df"} Oct 30 16:52:03.773363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:03.773221 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" event={"ID":"033a9d36-1eef-48f4-8615-f96fa569eee5","Type":"ContainerStarted","Data":"feb77340abe005af250cd298abdfdda6494110da6b7fcc5742827a3b19f27f3c"} Oct 30 16:52:03.773363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:03.773248 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" Oct 30 16:52:03.830599 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:03.830527 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" podStartSLOduration=1.8305094720000001 podStartE2EDuration="1.830509472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:03.827192729 +0000 UTC m=+1754.169866768" watchObservedRunningTime="2026-10-30 16:52:03.830509472 +0000 UTC m=+1754.173183511" Oct 30 16:52:04.464173 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:04.464140 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 16:52:04.464638 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:04.464197 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" podUID="b22f4e7a-ed39-41e3-9983-253a897bf603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 16:52:04.986103 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:04.986070 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb"] Oct 30 16:52:04.991274 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:04.991250 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.032748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.027704 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-4a8ab1-openshift-default-dockercfg-kbszs\"" Oct 30 16:52:05.045595 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.045550 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb"] Oct 30 16:52:05.135546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.135503 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-workload-certs\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.135945 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.135921 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-podinfo\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.136093 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.136079 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istiod-ca-cert\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.136230 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.136216 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk56w\" (UniqueName: \"kubernetes.io/projected/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-kube-api-access-xk56w\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.136347 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.136335 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-envoy\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.136566 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.136553 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-data\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.136708 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.136696 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-token\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.136874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.136860 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-workload-socket\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.137013 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.137001 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-credential-socket\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.238936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.238845 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-data\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.239858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.239833 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-token\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.240041 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.240025 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-workload-socket\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.240176 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.240161 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-credential-socket\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.240362 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.240347 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-workload-certs\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.240538 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.240522 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-podinfo\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.240671 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.240658 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istiod-ca-cert\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.281697 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.242839 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istiod-ca-cert\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.281956 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.243166 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-credential-socket\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.282089 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.243459 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-workload-certs\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.282209 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.239754 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-data\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.282320 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.244746 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-workload-socket\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.282435 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.247947 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-podinfo\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.282871 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.282796 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk56w\" (UniqueName: \"kubernetes.io/projected/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-kube-api-access-xk56w\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.283858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.283834 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-envoy\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.320756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.306613 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-envoy\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.320756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.310217 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-token\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.322461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.322406 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk56w\" (UniqueName: \"kubernetes.io/projected/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-kube-api-access-xk56w\") pod \"e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.341766 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.339108 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:05.465601 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.465155 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 16:52:05.465601 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.465230 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" podUID="b22f4e7a-ed39-41e3-9983-253a897bf603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 16:52:05.829974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.829946 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb"] Oct 30 16:52:05.830151 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.830019 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:05.830353 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.830332 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:05.830549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:05.830531 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:06.464253 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:06.464212 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 16:52:06.464458 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:06.464268 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" podUID="b22f4e7a-ed39-41e3-9983-253a897bf603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 16:52:06.864365 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:06.864324 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" event={"ID":"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d","Type":"ContainerStarted","Data":"ad95239a93ef7038f555175ae6f5c2f2a6057ff7cc68af02ca184706f06db97e"} Oct 30 16:52:06.864974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:06.864950 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" event={"ID":"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d","Type":"ContainerStarted","Data":"89352e9dde61baa244a01f8205dcab0ed332555f0841913e167ffbde8bf16210"} Oct 30 16:52:07.049226 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:07.049160 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" podStartSLOduration=3.049134608 podStartE2EDuration="3.049134608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:07.011257373 +0000 UTC m=+1757.353931426" watchObservedRunningTime="2026-10-30 16:52:07.049134608 +0000 UTC m=+1757.391808648" Oct 30 16:52:07.354761 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:07.348189 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:07.372383 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:07.372066 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:07.473519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:07.473032 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 16:52:07.473519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:07.473107 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" podUID="b22f4e7a-ed39-41e3-9983-253a897bf603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 16:52:07.653745 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:07.653590 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:07.756894 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:07.756850 4861 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:07.951273 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:07.951184 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:07.964755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:07.959816 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:07.965602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:07.965581 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:52:08.053753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:08.050483 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:08.366755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:08.364830 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:08.499750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:08.476220 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 16:52:08.500075 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:08.500027 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" podUID="b22f4e7a-ed39-41e3-9983-253a897bf603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 16:52:08.611098 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:08.609822 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:08.662298 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:08.659826 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:08.662298 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:08.660629 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:52:09.044529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:09.044491 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.061447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:09.061418 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:52:09.063037 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:09.062997 4861 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.108743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:09.108677 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.109333 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:09.109313 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:52:09.368660 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:09.368552 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.393805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:09.377208 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:52:09.577684 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:09.518096 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:52:09.577684 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:09.518162 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" podUID="16535ec9-31d5-4620-b9c9-2367ba14e26d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.613285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:09.581748 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 16:52:09.613285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:09.581808 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" podUID="b22f4e7a-ed39-41e3-9983-253a897bf603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 16:52:09.613285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:09.611998 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:09.613285 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:09.613244 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:52:09.698889 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:09.698787 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:10.103762 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:10.102169 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:52:10.143786 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:10.139106 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:10.366061 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:10.365963 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:52:10.394175 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:10.394127 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:10.394890 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:10.375669 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:10.474422 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:10.474385 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 16:52:10.507054 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:10.506919 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" podUID="b22f4e7a-ed39-41e3-9983-253a897bf603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 16:52:10.622007 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:10.621900 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:10.801707 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:10.801670 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 16:52:10.927152 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:10.927061 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:52:11.161750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:11.157220 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:11.364810 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:11.364682 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:52:11.480743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:11.475024 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 16:52:11.480743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:11.475084 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" podUID="b22f4e7a-ed39-41e3-9983-253a897bf603" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 16:52:11.490972 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:11.490095 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:52:11.494993 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:11.494967 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:52:11.496371 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:11.496351 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:52:13.182888 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:13.182858 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:13.183349 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:13.183311 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:13.184248 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:13.184227 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:52:14.781880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:14.781851 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" Oct 30 16:52:16.735428 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:16.735394 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k"] Oct 30 16:52:16.739980 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:16.739960 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:52:16.758457 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:16.758429 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-07cb3c-serving-cert\"" Oct 30 16:52:16.772918 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:16.772870 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k"] Oct 30 16:52:16.856212 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:16.856179 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9f9f36-9d38-42ae-a42b-014d740d3729-metrics-certs\") pod \"maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k\" (UID: \"1a9f9f36-9d38-42ae-a42b-014d740d3729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:52:16.856423 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:16.856231 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f9f36-9d38-42ae-a42b-014d740d3729-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k\" (UID: \"1a9f9f36-9d38-42ae-a42b-014d740d3729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:52:16.856423 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:16.856405 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlg9l\" (UniqueName: \"kubernetes.io/projected/1a9f9f36-9d38-42ae-a42b-014d740d3729-kube-api-access-jlg9l\") pod \"maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k\" (UID: \"1a9f9f36-9d38-42ae-a42b-014d740d3729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:52:16.856533 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:16.856454 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9f9f36-9d38-42ae-a42b-014d740d3729-maas-api-tls\") pod \"maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k\" (UID: \"1a9f9f36-9d38-42ae-a42b-014d740d3729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:52:16.972829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:16.971827 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9f9f36-9d38-42ae-a42b-014d740d3729-metrics-certs\") pod \"maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k\" (UID: \"1a9f9f36-9d38-42ae-a42b-014d740d3729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:52:16.972829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:16.971933 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f9f36-9d38-42ae-a42b-014d740d3729-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k\" (UID: \"1a9f9f36-9d38-42ae-a42b-014d740d3729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:52:16.972829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:16.972068 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlg9l\" (UniqueName: \"kubernetes.io/projected/1a9f9f36-9d38-42ae-a42b-014d740d3729-kube-api-access-jlg9l\") pod \"maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k\" (UID: \"1a9f9f36-9d38-42ae-a42b-014d740d3729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:52:16.972829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:16.972107 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9f9f36-9d38-42ae-a42b-014d740d3729-maas-api-tls\") pod \"maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k\" (UID: \"1a9f9f36-9d38-42ae-a42b-014d740d3729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:52:16.974016 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:16.973985 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f9f36-9d38-42ae-a42b-014d740d3729-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k\" (UID: \"1a9f9f36-9d38-42ae-a42b-014d740d3729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:52:16.992517 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:16.992401 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9f9f36-9d38-42ae-a42b-014d740d3729-metrics-certs\") pod \"maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k\" (UID: \"1a9f9f36-9d38-42ae-a42b-014d740d3729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:52:17.008556 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:17.008466 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9f9f36-9d38-42ae-a42b-014d740d3729-maas-api-tls\") pod \"maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k\" (UID: \"1a9f9f36-9d38-42ae-a42b-014d740d3729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:52:17.009446 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:17.009393 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlg9l\" (UniqueName: \"kubernetes.io/projected/1a9f9f36-9d38-42ae-a42b-014d740d3729-kube-api-access-jlg9l\") pod \"maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k\" (UID: \"1a9f9f36-9d38-42ae-a42b-014d740d3729\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:52:17.057904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:17.057869 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:52:17.381353 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:52:17.381309 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9f9f36_9d38_42ae_a42b_014d740d3729.slice/crio-71d70c34fdf35d3b5cadf59003c725c2ef5d843492fae1ff103063b64c94f4a3 WatchSource:0}: Error finding container 71d70c34fdf35d3b5cadf59003c725c2ef5d843492fae1ff103063b64c94f4a3: Status 404 returned error can't find the container with id 71d70c34fdf35d3b5cadf59003c725c2ef5d843492fae1ff103063b64c94f4a3 Oct 30 16:52:17.381838 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:17.381811 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k"] Oct 30 16:52:17.382932 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:17.382912 4861 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:52:17.452683 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:17.452650 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" event={"ID":"1a9f9f36-9d38-42ae-a42b-014d740d3729","Type":"ContainerStarted","Data":"71d70c34fdf35d3b5cadf59003c725c2ef5d843492fae1ff103063b64c94f4a3"} Oct 30 16:52:18.474162 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:18.474129 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" event={"ID":"1a9f9f36-9d38-42ae-a42b-014d740d3729","Type":"ContainerStarted","Data":"91992ca98c47ab3ec61d10c99e7e3befdad4dcb020247318ed1df2ee0a740e7b"} Oct 30 16:52:18.474582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:18.474234 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:52:18.521987 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:18.521920 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" podStartSLOduration=2.150342374 podStartE2EDuration="2.521902596s" totalImagesPullingTime="371.560222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:18.518361861 +0000 UTC m=+1768.861035910" watchObservedRunningTime="2026-10-30 16:52:18.521902596 +0000 UTC m=+1768.864576635" Oct 30 16:52:19.482046 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:19.482016 4861 generic.go:356] "Generic (PLEG): container finished" podID="963c434d-5cb8-4c34-bc1c-388e3cb1e125" containerID="1ec12ff8a0117350f2169865ef2a67e7d6d587e6fe51e906c512605d7cb5c01d" exitCode=0 Oct 30 16:52:19.482496 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:19.482093 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" event={"ID":"963c434d-5cb8-4c34-bc1c-388e3cb1e125","Type":"ContainerDied","Data":"1ec12ff8a0117350f2169865ef2a67e7d6d587e6fe51e906c512605d7cb5c01d"} Oct 30 16:52:19.482677 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:19.482660 4861 scope.go:122] "RemoveContainer" containerID="1ec12ff8a0117350f2169865ef2a67e7d6d587e6fe51e906c512605d7cb5c01d" Oct 30 16:52:19.484079 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:19.484052 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:19.484152 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:19.484120 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:19.484192 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:19.484150 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:20.290302 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:52:20.290255 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:52:20.490331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:20.490290 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" event={"ID":"963c434d-5cb8-4c34-bc1c-388e3cb1e125","Type":"ContainerStarted","Data":"68b7249d6114c332b25b1148e3da19b0920dd8cfc835a5c520d0e8e334a943ad"} Oct 30 16:52:21.173280 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:21.173249 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:52:21.173463 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:21.173418 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 16:52:21.173512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:21.173465 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" podUID="963c434d-5cb8-4c34-bc1c-388e3cb1e125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 16:52:22.172977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:22.172938 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 16:52:22.173424 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:22.172995 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" podUID="963c434d-5cb8-4c34-bc1c-388e3cb1e125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 16:52:23.173469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:23.173436 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 16:52:23.174004 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:23.173500 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" podUID="963c434d-5cb8-4c34-bc1c-388e3cb1e125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 16:52:24.172943 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:24.172911 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 16:52:24.173138 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:24.172959 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" podUID="963c434d-5cb8-4c34-bc1c-388e3cb1e125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 16:52:24.488008 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:24.487981 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:52:25.173261 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:25.173222 4861 patch_prober.go:29] interesting pod/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 30 16:52:25.173449 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:25.173291 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" podUID="963c434d-5cb8-4c34-bc1c-388e3cb1e125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 30 16:52:26.178385 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:26.178352 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:52:26.178892 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:26.178756 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:52:26.179417 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:26.179400 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:52:30.148294 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.148236 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:30.240748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.228973 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4"] Oct 30 16:52:30.268755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.249189 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.305755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.273425 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4"] Oct 30 16:52:30.419513 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.419476 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.424746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.419822 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.424746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.419877 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwddr\" (UniqueName: \"kubernetes.io/projected/310bed42-3cfb-4b4d-b075-de011a27909f-kube-api-access-lwddr\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.424746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.419943 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.424746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.420016 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310bed42-3cfb-4b4d-b075-de011a27909f-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.424746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.420111 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.424746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.420217 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310bed42-3cfb-4b4d-b075-de011a27909f-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.424746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.420313 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.424746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.420344 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310bed42-3cfb-4b4d-b075-de011a27909f-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.464748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.456146 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:30.568750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.568423 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.568750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.568493 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.568750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.568530 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwddr\" (UniqueName: \"kubernetes.io/projected/310bed42-3cfb-4b4d-b075-de011a27909f-kube-api-access-lwddr\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.568750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.568593 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.568750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.568649 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310bed42-3cfb-4b4d-b075-de011a27909f-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.568750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.568674 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.569262 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.568768 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310bed42-3cfb-4b4d-b075-de011a27909f-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.569262 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.568856 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.569262 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.568887 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310bed42-3cfb-4b4d-b075-de011a27909f-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.577747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.569796 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310bed42-3cfb-4b4d-b075-de011a27909f-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.577747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.570099 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.604747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.604240 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.604747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.604522 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.613784 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.605009 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.624799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.624695 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310bed42-3cfb-4b4d-b075-de011a27909f-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.640522 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.640495 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.667967 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.665953 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310bed42-3cfb-4b4d-b075-de011a27909f-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.694753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.670822 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwddr\" (UniqueName: \"kubernetes.io/projected/310bed42-3cfb-4b4d-b075-de011a27909f-kube-api-access-lwddr\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:30.880768 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:30.878036 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:31.310409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:31.310378 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4"] Oct 30 16:52:31.312667 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:52:31.312642 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310bed42_3cfb_4b4d_b075_de011a27909f.slice/crio-2bcb22d5261c562f13a71d9f99ca425103a99725185816c5b5de9661c665b799 WatchSource:0}: Error finding container 2bcb22d5261c562f13a71d9f99ca425103a99725185816c5b5de9661c665b799: Status 404 returned error can't find the container with id 2bcb22d5261c562f13a71d9f99ca425103a99725185816c5b5de9661c665b799 Oct 30 16:52:31.315845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:31.315804 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:31.315968 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:31.315890 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:31.315968 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:31.315933 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:31.808493 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:31.808460 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" event={"ID":"310bed42-3cfb-4b4d-b075-de011a27909f","Type":"ContainerStarted","Data":"c80ddde3acd7e2e7fd3fe54cd825da9aaaa133b7c426de6cf250e9b77cb67ea6"} Oct 30 16:52:31.808698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:31.808511 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" event={"ID":"310bed42-3cfb-4b4d-b075-de011a27909f","Type":"ContainerStarted","Data":"2bcb22d5261c562f13a71d9f99ca425103a99725185816c5b5de9661c665b799"} Oct 30 16:52:31.857912 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:31.857864 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" podStartSLOduration=1.85785226 podStartE2EDuration="1.85785226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:31.855365291 +0000 UTC m=+1782.198039329" watchObservedRunningTime="2026-10-30 16:52:31.85785226 +0000 UTC m=+1782.200526299" Oct 30 16:52:31.881214 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:31.881177 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:32.883296 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:32.883263 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:52:32.883674 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:32.883322 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" podUID="310bed42-3cfb-4b4d-b075-de011a27909f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 16:52:33.882027 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:33.881991 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:52:33.882220 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:33.882056 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" podUID="310bed42-3cfb-4b4d-b075-de011a27909f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 16:52:34.881049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:34.881009 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:52:34.881511 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:34.881079 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" podUID="310bed42-3cfb-4b4d-b075-de011a27909f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 16:52:35.881662 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:35.881625 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:52:35.882090 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:35.881677 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" podUID="310bed42-3cfb-4b4d-b075-de011a27909f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 16:52:36.881494 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:36.881461 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:52:36.881700 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:36.881517 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" podUID="310bed42-3cfb-4b4d-b075-de011a27909f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 16:52:37.937891 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:37.937854 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:52:37.938582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:37.938549 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" podUID="310bed42-3cfb-4b4d-b075-de011a27909f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 16:52:38.881941 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:38.881906 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 30 16:52:38.882106 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:38.881967 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" podUID="310bed42-3cfb-4b4d-b075-de011a27909f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 30 16:52:39.885937 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:39.885902 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:39.886339 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:39.886189 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:39.887034 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:39.887012 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:52:42.375297 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.375268 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl"] Oct 30 16:52:42.379798 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.379771 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.399151 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.399111 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-77f4f4-openshift-default-dockercfg-bkbhp\"" Oct 30 16:52:42.407239 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.407187 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rt25\" (UniqueName: \"kubernetes.io/projected/db2f1614-1fc7-437b-89c8-001c8f180877-kube-api-access-9rt25\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.407401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.407297 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2f1614-1fc7-437b-89c8-001c8f180877-istio-token\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.407401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.407397 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-istio-envoy\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.407541 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.407441 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-istio-data\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.407541 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.407496 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2f1614-1fc7-437b-89c8-001c8f180877-istiod-ca-cert\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.407541 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.407533 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-workload-socket\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.407795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.407569 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-credential-socket\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.407795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.407593 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2f1614-1fc7-437b-89c8-001c8f180877-istio-podinfo\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.407795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.407712 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-workload-certs\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.412879 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.412857 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl"] Oct 30 16:52:42.559930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.531591 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-workload-certs\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.560299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.560272 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rt25\" (UniqueName: \"kubernetes.io/projected/db2f1614-1fc7-437b-89c8-001c8f180877-kube-api-access-9rt25\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.561651 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.561625 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2f1614-1fc7-437b-89c8-001c8f180877-istio-token\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.561941 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.561919 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-istio-envoy\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.562113 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.562093 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-istio-data\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.562289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.562269 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2f1614-1fc7-437b-89c8-001c8f180877-istiod-ca-cert\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.562467 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.562448 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-workload-socket\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.562622 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.562605 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-credential-socket\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.562761 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.562744 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2f1614-1fc7-437b-89c8-001c8f180877-istio-podinfo\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.576754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.566064 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-istio-data\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.576754 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.535223 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-workload-certs\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.577395 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.577366 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-workload-socket\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.579698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.578136 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-credential-socket\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.579873 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.578283 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2f1614-1fc7-437b-89c8-001c8f180877-istiod-ca-cert\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.594049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.594014 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-istio-envoy\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.597314 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.597287 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2f1614-1fc7-437b-89c8-001c8f180877-istio-token\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.608032 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.608007 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2f1614-1fc7-437b-89c8-001c8f180877-istio-podinfo\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.608808 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.608756 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rt25\" (UniqueName: \"kubernetes.io/projected/db2f1614-1fc7-437b-89c8-001c8f180877-kube-api-access-9rt25\") pod \"e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:42.701049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:42.700943 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:43.037956 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:43.037924 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl"] Oct 30 16:52:43.039167 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:52:43.039134 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2f1614_1fc7_437b_89c8_001c8f180877.slice/crio-f973f4c2ecd3ef985b5ab0cde119f50431ed09d16aa5bff70dc99c06c2ac608c WatchSource:0}: Error finding container f973f4c2ecd3ef985b5ab0cde119f50431ed09d16aa5bff70dc99c06c2ac608c: Status 404 returned error can't find the container with id f973f4c2ecd3ef985b5ab0cde119f50431ed09d16aa5bff70dc99c06c2ac608c Oct 30 16:52:43.041773 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:43.041704 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:43.041842 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:43.041830 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:43.041883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:43.041865 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:43.947877 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:43.947842 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" event={"ID":"db2f1614-1fc7-437b-89c8-001c8f180877","Type":"ContainerStarted","Data":"898a23a78e9f5fdabab7bee16d8bbc1ce72c5364ba30b91a0c4a64de0430918c"} Oct 30 16:52:43.948261 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:43.947901 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" event={"ID":"db2f1614-1fc7-437b-89c8-001c8f180877","Type":"ContainerStarted","Data":"f973f4c2ecd3ef985b5ab0cde119f50431ed09d16aa5bff70dc99c06c2ac608c"} Oct 30 16:52:44.013415 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:44.013366 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" podStartSLOduration=2.013352427 podStartE2EDuration="2.013352427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:44.009694364 +0000 UTC m=+1794.352368402" watchObservedRunningTime="2026-10-30 16:52:44.013352427 +0000 UTC m=+1794.356026464" Oct 30 16:52:44.703436 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:44.703402 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:44.710366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:44.710341 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:44.954818 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:44.954747 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:44.955854 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:44.955828 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:52:45.782508 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:45.782471 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc"] Oct 30 16:52:45.798537 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:45.798500 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:45.821701 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:45.821669 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc"] Oct 30 16:52:45.933120 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:45.933087 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:45.933120 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:45.933121 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:45.933351 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:45.933142 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:45.933351 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:45.933216 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:45.933351 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:45.933303 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:45.933496 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:45.933368 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:45.933496 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:45.933443 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:45.933634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:45.933609 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:45.933763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:45.933662 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l82g\" (UniqueName: \"kubernetes.io/projected/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-kube-api-access-7l82g\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.035091 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.035016 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.035091 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.035061 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.035863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.035186 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.035863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.035222 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l82g\" (UniqueName: \"kubernetes.io/projected/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-kube-api-access-7l82g\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.035863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.035310 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.035863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.035330 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.035863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.035346 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.035863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.035369 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.035863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.035402 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.035863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.035466 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-credential-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.035863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.035691 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-workload-socket\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.035863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.035810 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-data\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.035863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.035838 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-workload-certs\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.035863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.035851 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istiod-ca-cert\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.037900 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.037878 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-podinfo\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.038003 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.037903 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-envoy\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.059542 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.059517 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-token\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.059638 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.059590 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l82g\" (UniqueName: \"kubernetes.io/projected/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-kube-api-access-7l82g\") pod \"e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.121026 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.121002 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:46.448450 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.448418 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc"] Oct 30 16:52:46.449278 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:52:46.449251 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcdcc450_d8cb_4d93_b4ae_0ed5dff857f2.slice/crio-6ae63ea72ded0a2ff42e408bb8e9393c0fed950fefde54dc7903c62b7844d6b5 WatchSource:0}: Error finding container 6ae63ea72ded0a2ff42e408bb8e9393c0fed950fefde54dc7903c62b7844d6b5: Status 404 returned error can't find the container with id 6ae63ea72ded0a2ff42e408bb8e9393c0fed950fefde54dc7903c62b7844d6b5 Oct 30 16:52:46.451982 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.451949 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:46.452105 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.452026 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:46.452105 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.452069 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:52:46.969717 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.969675 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" event={"ID":"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2","Type":"ContainerStarted","Data":"20ac9166683516cc981d646eb5e5859f896a853a7563fba4ceb2c6c133ccd392"} Oct 30 16:52:46.969936 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:46.969759 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" event={"ID":"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2","Type":"ContainerStarted","Data":"6ae63ea72ded0a2ff42e408bb8e9393c0fed950fefde54dc7903c62b7844d6b5"} Oct 30 16:52:47.033021 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:47.032976 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" podStartSLOduration=2.032961981 podStartE2EDuration="2.032961981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:47.02979128 +0000 UTC m=+1797.372465319" watchObservedRunningTime="2026-10-30 16:52:47.032961981 +0000 UTC m=+1797.375636019" Oct 30 16:52:47.121299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:47.121266 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:47.122759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:47.122704 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 16:52:47.122889 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:47.122780 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" podUID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 16:52:48.122943 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:48.122901 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 16:52:48.123614 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:48.123569 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" podUID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 16:52:48.645219 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:48.645189 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9"] Oct 30 16:52:48.649963 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:48.649937 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" Oct 30 16:52:48.672708 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:48.672678 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9"] Oct 30 16:52:48.744338 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:48.743813 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-77f4f4-dockercfg-qsbjd\"" Oct 30 16:52:48.744338 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:48.743883 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-77f4f4\"" Oct 30 16:52:48.998942 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:48.998018 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d066599-937f-44b9-89d0-dadc7d7478e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9\" (UID: \"5d066599-937f-44b9-89d0-dadc7d7478e9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" Oct 30 16:52:48.998942 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:48.998273 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptbs2\" (UniqueName: \"kubernetes.io/projected/5d066599-937f-44b9-89d0-dadc7d7478e9-kube-api-access-ptbs2\") pod \"payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9\" (UID: \"5d066599-937f-44b9-89d0-dadc7d7478e9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" Oct 30 16:52:49.011746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.009842 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m"] Oct 30 16:52:49.025747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.015661 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" Oct 30 16:52:49.025747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.025639 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m"] Oct 30 16:52:49.099406 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.099366 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptbs2\" (UniqueName: \"kubernetes.io/projected/5d066599-937f-44b9-89d0-dadc7d7478e9-kube-api-access-ptbs2\") pod \"payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9\" (UID: \"5d066599-937f-44b9-89d0-dadc7d7478e9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" Oct 30 16:52:49.099840 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.099815 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d066599-937f-44b9-89d0-dadc7d7478e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9\" (UID: \"5d066599-937f-44b9-89d0-dadc7d7478e9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" Oct 30 16:52:49.100448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.100422 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d066599-937f-44b9-89d0-dadc7d7478e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9\" (UID: \"5d066599-937f-44b9-89d0-dadc7d7478e9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" Oct 30 16:52:49.126938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.126776 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 16:52:49.126938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.126830 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" podUID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 16:52:49.129709 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.129681 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptbs2\" (UniqueName: \"kubernetes.io/projected/5d066599-937f-44b9-89d0-dadc7d7478e9-kube-api-access-ptbs2\") pod \"payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9\" (UID: \"5d066599-937f-44b9-89d0-dadc7d7478e9\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" Oct 30 16:52:49.224747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.224322 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d0fc53-fb6b-4437-a486-bb8ab2e3e39a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m\" (UID: \"81d0fc53-fb6b-4437-a486-bb8ab2e3e39a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" Oct 30 16:52:49.224747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.224416 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf67s\" (UniqueName: \"kubernetes.io/projected/81d0fc53-fb6b-4437-a486-bb8ab2e3e39a-kube-api-access-qf67s\") pod \"payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m\" (UID: \"81d0fc53-fb6b-4437-a486-bb8ab2e3e39a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" Oct 30 16:52:49.266985 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.266919 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" Oct 30 16:52:49.333528 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.333488 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d0fc53-fb6b-4437-a486-bb8ab2e3e39a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m\" (UID: \"81d0fc53-fb6b-4437-a486-bb8ab2e3e39a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" Oct 30 16:52:49.333691 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.333549 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf67s\" (UniqueName: \"kubernetes.io/projected/81d0fc53-fb6b-4437-a486-bb8ab2e3e39a-kube-api-access-qf67s\") pod \"payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m\" (UID: \"81d0fc53-fb6b-4437-a486-bb8ab2e3e39a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" Oct 30 16:52:49.334446 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.334425 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d0fc53-fb6b-4437-a486-bb8ab2e3e39a-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m\" (UID: \"81d0fc53-fb6b-4437-a486-bb8ab2e3e39a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" Oct 30 16:52:49.363299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.363266 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf67s\" (UniqueName: \"kubernetes.io/projected/81d0fc53-fb6b-4437-a486-bb8ab2e3e39a-kube-api-access-qf67s\") pod \"payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m\" (UID: \"81d0fc53-fb6b-4437-a486-bb8ab2e3e39a\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" Oct 30 16:52:49.596479 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.596453 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9"] Oct 30 16:52:49.597101 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:52:49.597069 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d066599_937f_44b9_89d0_dadc7d7478e9.slice/crio-e5aa17cb2370dace6250615212edaf2d73ca33bce2b80586442ed3c061a57381 WatchSource:0}: Error finding container e5aa17cb2370dace6250615212edaf2d73ca33bce2b80586442ed3c061a57381: Status 404 returned error can't find the container with id e5aa17cb2370dace6250615212edaf2d73ca33bce2b80586442ed3c061a57381 Oct 30 16:52:49.639759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.639736 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" Oct 30 16:52:49.958455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:49.958428 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m"] Oct 30 16:52:49.960124 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:52:49.960092 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d0fc53_fb6b_4437_a486_bb8ab2e3e39a.slice/crio-0c6ae7a15c14b3736b3a4f3efb55a7442cdc26e5e4544402a0923cba2b1c2e84 WatchSource:0}: Error finding container 0c6ae7a15c14b3736b3a4f3efb55a7442cdc26e5e4544402a0923cba2b1c2e84: Status 404 returned error can't find the container with id 0c6ae7a15c14b3736b3a4f3efb55a7442cdc26e5e4544402a0923cba2b1c2e84 Oct 30 16:52:50.039896 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:50.039849 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" event={"ID":"5d066599-937f-44b9-89d0-dadc7d7478e9","Type":"ContainerStarted","Data":"25e082e239af41b617a26f28a0ea7d4213bf497f083660fd530ea406ce4478b2"} Oct 30 16:52:50.040061 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:50.039919 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" event={"ID":"5d066599-937f-44b9-89d0-dadc7d7478e9","Type":"ContainerStarted","Data":"e5aa17cb2370dace6250615212edaf2d73ca33bce2b80586442ed3c061a57381"} Oct 30 16:52:50.040061 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:50.039966 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" Oct 30 16:52:50.041533 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:50.041509 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" event={"ID":"81d0fc53-fb6b-4437-a486-bb8ab2e3e39a","Type":"ContainerStarted","Data":"0c6ae7a15c14b3736b3a4f3efb55a7442cdc26e5e4544402a0923cba2b1c2e84"} Oct 30 16:52:50.084860 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:50.084840 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:52:50.087682 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:50.087490 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" podStartSLOduration=2.087476609 podStartE2EDuration="2.087476609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:50.083236701 +0000 UTC m=+1800.425910738" watchObservedRunningTime="2026-10-30 16:52:50.087476609 +0000 UTC m=+1800.430150678" Oct 30 16:52:50.090669 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:50.090648 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:52:50.126789 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:50.126754 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 16:52:50.126957 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:50.126834 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" podUID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 16:52:50.354572 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:52:50.354500 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:52:51.050782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:51.050753 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" event={"ID":"81d0fc53-fb6b-4437-a486-bb8ab2e3e39a","Type":"ContainerStarted","Data":"b317c83873850bb81d69281275c4399e9c7db1b64a84c07a59a0ce5959c30a88"} Oct 30 16:52:51.051154 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:51.051126 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" Oct 30 16:52:51.128829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:51.127923 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 16:52:51.128829 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:51.127990 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" podUID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 16:52:52.125760 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:52.125074 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 16:52:52.125760 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:52.125134 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" podUID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 16:52:53.134848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:53.132649 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 16:52:53.134848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:53.132744 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" podUID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 16:52:54.205749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:54.173284 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 16:52:54.205749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:54.173383 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" podUID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 16:52:54.242895 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:54.208849 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" podStartSLOduration=6.208827639 podStartE2EDuration="6.208827639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:51.127625191 +0000 UTC m=+1801.470299227" watchObservedRunningTime="2026-10-30 16:52:54.208827639 +0000 UTC m=+1804.551501680" Oct 30 16:52:54.303749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:54.292736 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr"] Oct 30 16:52:54.340757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:54.330388 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" Oct 30 16:52:54.377751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:54.342179 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr"] Oct 30 16:52:54.492195 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:54.492157 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-cb6251-dockercfg-r74wg\"" Oct 30 16:52:54.493208 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:54.493054 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-cb6251\"" Oct 30 16:52:54.495941 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:54.494785 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 16:52:54.704578 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:54.627676 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7bwx\" (UniqueName: \"kubernetes.io/projected/b4ce5954-4591-4f16-84d5-36f11350dffc-kube-api-access-q7bwx\") pod \"payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr\" (UID: \"b4ce5954-4591-4f16-84d5-36f11350dffc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" Oct 30 16:52:54.708747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:54.705067 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ce5954-4591-4f16-84d5-36f11350dffc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr\" (UID: \"b4ce5954-4591-4f16-84d5-36f11350dffc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" Oct 30 16:52:54.738750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:54.714092 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 16:52:54.823805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:54.823060 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7bwx\" (UniqueName: \"kubernetes.io/projected/b4ce5954-4591-4f16-84d5-36f11350dffc-kube-api-access-q7bwx\") pod \"payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr\" (UID: \"b4ce5954-4591-4f16-84d5-36f11350dffc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" Oct 30 16:52:54.823805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:54.823227 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ce5954-4591-4f16-84d5-36f11350dffc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr\" (UID: \"b4ce5954-4591-4f16-84d5-36f11350dffc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" Oct 30 16:52:54.854509 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:54.854474 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ce5954-4591-4f16-84d5-36f11350dffc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr\" (UID: \"b4ce5954-4591-4f16-84d5-36f11350dffc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" Oct 30 16:52:54.990740 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:54.990683 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7bwx\" (UniqueName: \"kubernetes.io/projected/b4ce5954-4591-4f16-84d5-36f11350dffc-kube-api-access-q7bwx\") pod \"payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr\" (UID: \"b4ce5954-4591-4f16-84d5-36f11350dffc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" Oct 30 16:52:55.095810 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:55.086642 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" Oct 30 16:52:55.122114 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:55.122075 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 16:52:55.122295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:55.122154 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" podUID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 16:52:55.142901 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:55.142399 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:55.267885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:55.245833 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:55.458971 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:55.458871 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 16:52:55.506763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:55.499813 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:55.506763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:55.500284 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:52:55.807757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:55.776833 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 16:52:55.807757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:55.777695 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:52:55.898904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:55.898867 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:52:55.899240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:55.899187 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" podUID="84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:55.978835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:55.978655 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:52:55.979226 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:55.979171 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" podUID="310bed42-3cfb-4b4d-b075-de011a27909f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:56.188897 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:56.188854 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 16:52:56.189194 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:56.189147 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" podUID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 16:52:56.353246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:56.218648 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:56.400753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:56.257935 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:56.401673 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:56.354840 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:52:56.402163 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:56.402146 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:52:56.661801 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:56.657541 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 16:52:56.661801 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:56.660423 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:52:56.804262 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:56.804207 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Oct 30 16:52:57.122189 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:56.894629 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:57.122189 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:56.907629 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:52:57.122189 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:57.051969 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" podUID="f9951912-7c87-4ed5-8525-073105af8c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:57.461701 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:57.459903 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 16:52:57.610781 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:57.610179 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 16:52:57.828381 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:57.824382 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded" Oct 30 16:52:57.828381 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:57.826793 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:57.899893 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:57.897977 4861 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:58.278750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:58.275098 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:58.327790 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:58.281864 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr"] Oct 30 16:52:58.328111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:58.299178 4861 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Oct 30 16:52:58.378415 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:58.376864 4861 patch_prober.go:29] interesting pod/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:52:58.378415 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:58.376928 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" podUID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:58.470752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:58.462176 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:58.637884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:58.613355 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:52:58.649759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:58.617439 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" Oct 30 16:52:58.651479 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:58.651452 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 16:52:58.692199 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:58.692172 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 16:52:58.699767 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:58.699243 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 16:52:58.701234 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:58.701203 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" event={"ID":"b4ce5954-4591-4f16-84d5-36f11350dffc","Type":"ContainerStarted","Data":"1d5291d07c921cee5b7dec26a0a843a3ead0bb4b2a8e1198a5f6435cdf9061ae"} Oct 30 16:52:58.712819 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:58.703759 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 16:52:59.032586 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:59.032546 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:59.032989 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:59.032964 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:59.033828 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:59.033802 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 16:52:59.740741 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:59.740098 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" event={"ID":"b4ce5954-4591-4f16-84d5-36f11350dffc","Type":"ContainerStarted","Data":"e70aff4f8751ae00b5c3830ff5ef1bd57ebce5998f1ff9188911da4a4d08b9b3"} Oct 30 16:52:59.740741 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:59.740159 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" Oct 30 16:52:59.797980 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:52:59.797929 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" podStartSLOduration=5.797915456 podStartE2EDuration="5.797915456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:59.782422091 +0000 UTC m=+1810.125096130" watchObservedRunningTime="2026-10-30 16:52:59.797915456 +0000 UTC m=+1810.140589494" Oct 30 16:53:01.054325 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:01.054297 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" Oct 30 16:53:01.799344 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:01.799313 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5"] Oct 30 16:53:01.831982 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:01.831952 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" Oct 30 16:53:01.844065 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:01.844037 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5"] Oct 30 16:53:01.848078 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:01.848054 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-dd715e\"" Oct 30 16:53:01.848201 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:01.848131 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-dd715e-dockercfg-9fdcv\"" Oct 30 16:53:01.969751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:01.969441 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c8b7b8-b899-4303-870f-dce7e002fb42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5\" (UID: \"26c8b7b8-b899-4303-870f-dce7e002fb42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" Oct 30 16:53:01.969965 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:01.969793 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkqd9\" (UniqueName: \"kubernetes.io/projected/26c8b7b8-b899-4303-870f-dce7e002fb42-kube-api-access-mkqd9\") pod \"payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5\" (UID: \"26c8b7b8-b899-4303-870f-dce7e002fb42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" Oct 30 16:53:02.151142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.151060 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" Oct 30 16:53:02.158585 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.156043 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c8b7b8-b899-4303-870f-dce7e002fb42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5\" (UID: \"26c8b7b8-b899-4303-870f-dce7e002fb42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" Oct 30 16:53:02.159058 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.159030 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkqd9\" (UniqueName: \"kubernetes.io/projected/26c8b7b8-b899-4303-870f-dce7e002fb42-kube-api-access-mkqd9\") pod \"payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5\" (UID: \"26c8b7b8-b899-4303-870f-dce7e002fb42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" Oct 30 16:53:02.160646 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.160621 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c8b7b8-b899-4303-870f-dce7e002fb42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5\" (UID: \"26c8b7b8-b899-4303-870f-dce7e002fb42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" Oct 30 16:53:02.184787 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.182818 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj"] Oct 30 16:53:02.299229 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.299188 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkqd9\" (UniqueName: \"kubernetes.io/projected/26c8b7b8-b899-4303-870f-dce7e002fb42-kube-api-access-mkqd9\") pod \"payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5\" (UID: \"26c8b7b8-b899-4303-870f-dce7e002fb42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" Oct 30 16:53:02.319752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.300974 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" Oct 30 16:53:02.354271 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.354239 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj"] Oct 30 16:53:02.453380 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.453300 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" Oct 30 16:53:02.460257 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.457068 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5k5s\" (UniqueName: \"kubernetes.io/projected/cff713aa-db97-4366-ae04-c2bb57261598-kube-api-access-z5k5s\") pod \"payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj\" (UID: \"cff713aa-db97-4366-ae04-c2bb57261598\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" Oct 30 16:53:02.460257 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.457290 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff713aa-db97-4366-ae04-c2bb57261598-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj\" (UID: \"cff713aa-db97-4366-ae04-c2bb57261598\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" Oct 30 16:53:02.558676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.558642 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5k5s\" (UniqueName: \"kubernetes.io/projected/cff713aa-db97-4366-ae04-c2bb57261598-kube-api-access-z5k5s\") pod \"payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj\" (UID: \"cff713aa-db97-4366-ae04-c2bb57261598\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" Oct 30 16:53:02.558885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.558803 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff713aa-db97-4366-ae04-c2bb57261598-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj\" (UID: \"cff713aa-db97-4366-ae04-c2bb57261598\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" Oct 30 16:53:02.559602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.559485 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff713aa-db97-4366-ae04-c2bb57261598-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj\" (UID: \"cff713aa-db97-4366-ae04-c2bb57261598\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" Oct 30 16:53:02.772107 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.771931 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5k5s\" (UniqueName: \"kubernetes.io/projected/cff713aa-db97-4366-ae04-c2bb57261598-kube-api-access-z5k5s\") pod \"payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj\" (UID: \"cff713aa-db97-4366-ae04-c2bb57261598\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" Oct 30 16:53:02.832171 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.832143 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5"] Oct 30 16:53:02.834694 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:53:02.834043 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c8b7b8_b899_4303_870f_dce7e002fb42.slice/crio-b0702e038efb2d272e2e6b3068e2adbc16ccddce57ffe306c44062d12b3c6977 WatchSource:0}: Error finding container b0702e038efb2d272e2e6b3068e2adbc16ccddce57ffe306c44062d12b3c6977: Status 404 returned error can't find the container with id b0702e038efb2d272e2e6b3068e2adbc16ccddce57ffe306c44062d12b3c6977 Oct 30 16:53:02.971145 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:02.971110 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" Oct 30 16:53:03.316765 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:53:03.316185 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff713aa_db97_4366_ae04_c2bb57261598.slice/crio-2a4cd9099533727e77eff2a9baa064647c3e8e710f40c215d0e1bde65116ebaa WatchSource:0}: Error finding container 2a4cd9099533727e77eff2a9baa064647c3e8e710f40c215d0e1bde65116ebaa: Status 404 returned error can't find the container with id 2a4cd9099533727e77eff2a9baa064647c3e8e710f40c215d0e1bde65116ebaa Oct 30 16:53:03.317308 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:03.317276 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj"] Oct 30 16:53:03.792938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:03.792903 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" event={"ID":"cff713aa-db97-4366-ae04-c2bb57261598","Type":"ContainerStarted","Data":"8d93b706b96f780e2f683a349de87f3ca93b7916ef7c7d0732adbab88199a671"} Oct 30 16:53:03.793182 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:03.792966 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" event={"ID":"cff713aa-db97-4366-ae04-c2bb57261598","Type":"ContainerStarted","Data":"2a4cd9099533727e77eff2a9baa064647c3e8e710f40c215d0e1bde65116ebaa"} Oct 30 16:53:03.793182 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:03.792998 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" Oct 30 16:53:03.794904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:03.794875 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" event={"ID":"26c8b7b8-b899-4303-870f-dce7e002fb42","Type":"ContainerStarted","Data":"b44ce697cbe329b4ec99d74e148212d8df50a00b4a8f7c93c528406d9e737dc6"} Oct 30 16:53:03.795046 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:03.794931 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" Oct 30 16:53:03.795046 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:03.794950 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" event={"ID":"26c8b7b8-b899-4303-870f-dce7e002fb42","Type":"ContainerStarted","Data":"b0702e038efb2d272e2e6b3068e2adbc16ccddce57ffe306c44062d12b3c6977"} Oct 30 16:53:03.835644 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:03.835584 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" podStartSLOduration=2.835571489 podStartE2EDuration="2.835571489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:03.833854514 +0000 UTC m=+1814.176528553" watchObservedRunningTime="2026-10-30 16:53:03.835571489 +0000 UTC m=+1814.178245527" Oct 30 16:53:03.875840 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:03.875797 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" podStartSLOduration=2.875784616 podStartE2EDuration="2.875784616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:03.875106784 +0000 UTC m=+1814.217780824" watchObservedRunningTime="2026-10-30 16:53:03.875784616 +0000 UTC m=+1814.218458654" Oct 30 16:53:07.823503 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:07.823469 4861 generic.go:356] "Generic (PLEG): container finished" podID="0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" containerID="ad95239a93ef7038f555175ae6f5c2f2a6057ff7cc68af02ca184706f06db97e" exitCode=0 Oct 30 16:53:07.824001 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:07.823548 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" event={"ID":"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d","Type":"ContainerDied","Data":"ad95239a93ef7038f555175ae6f5c2f2a6057ff7cc68af02ca184706f06db97e"} Oct 30 16:53:07.824001 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:07.823989 4861 scope.go:122] "RemoveContainer" containerID="ad95239a93ef7038f555175ae6f5c2f2a6057ff7cc68af02ca184706f06db97e" Oct 30 16:53:07.825566 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:07.825538 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:53:07.825659 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:07.825608 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:53:07.825659 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:07.825639 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:53:07.951063 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:07.951034 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:53:08.833287 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:08.833252 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" event={"ID":"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d","Type":"ContainerStarted","Data":"e06afb8db231036a8e207efeb3eceafaca53c5c63f3db41b9e9fad53da81d9f1"} Oct 30 16:53:09.343059 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:09.343029 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:53:09.344556 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:09.344530 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 16:53:09.344690 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:09.344575 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" podUID="0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 16:53:10.346786 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:10.343153 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 16:53:10.346786 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:10.343218 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" podUID="0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 16:53:10.750046 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:10.750012 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" Oct 30 16:53:11.342884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:11.342849 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 16:53:11.343177 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:11.343145 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" podUID="0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 16:53:12.342455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:12.342419 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 16:53:12.342877 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:12.342493 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" podUID="0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 16:53:13.345122 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:13.345084 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 16:53:13.345706 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:13.345674 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" podUID="0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 16:53:14.351185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:14.351143 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 16:53:14.351810 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:14.351775 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" podUID="0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 16:53:14.830182 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:14.830149 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" Oct 30 16:53:14.842925 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:14.842882 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" Oct 30 16:53:15.529867 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:15.529818 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:53:15.758107 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:15.758080 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:53:15.759894 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:15.759825 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:53:15.769129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:15.768923 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:53:16.568817 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.568787 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd"] Oct 30 16:53:16.574443 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.574420 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.599569 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.599532 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd"] Oct 30 16:53:16.605463 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.605442 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-cf4621-openshift-default-dockercfg-5q9dp\"" Oct 30 16:53:16.666216 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.666183 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-workload-socket\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.666216 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.666218 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-workload-certs\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.666439 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.666254 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-token\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.666439 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.666355 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e4e14f-88e4-4fac-a6fd-04baae40e066-istiod-ca-cert\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.666439 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.666382 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-envoy\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.666600 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.666559 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-data\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.666663 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.666617 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-credential-socket\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.666739 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.666663 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lw78\" (UniqueName: \"kubernetes.io/projected/41e4e14f-88e4-4fac-a6fd-04baae40e066-kube-api-access-4lw78\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.666857 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.666841 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-podinfo\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.779747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.779681 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-data\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.780608 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.780584 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-credential-socket\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.780859 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.780841 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lw78\" (UniqueName: \"kubernetes.io/projected/41e4e14f-88e4-4fac-a6fd-04baae40e066-kube-api-access-4lw78\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.781128 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.781112 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-podinfo\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.781274 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.781260 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-workload-socket\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.781384 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.781369 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-workload-certs\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.781501 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.781488 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-token\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.781619 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.781604 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e4e14f-88e4-4fac-a6fd-04baae40e066-istiod-ca-cert\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.781750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.781715 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-envoy\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.782877 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.782853 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-credential-socket\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.783126 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.780505 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-data\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.791955 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.791931 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-workload-certs\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.793066 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.792894 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e4e14f-88e4-4fac-a6fd-04baae40e066-istiod-ca-cert\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.793219 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.793007 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-workload-socket\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.811089 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.811056 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-envoy\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.812138 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.812112 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-podinfo\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.813966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.813647 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lw78\" (UniqueName: \"kubernetes.io/projected/41e4e14f-88e4-4fac-a6fd-04baae40e066-kube-api-access-4lw78\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.832935 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.832858 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-token\") pod \"e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:16.897637 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:16.897606 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:17.234207 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.234175 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf"] Oct 30 16:53:17.238824 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.238804 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:17.248772 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.248746 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd"] Oct 30 16:53:17.250536 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:53:17.250508 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e4e14f_88e4_4fac_a6fd_04baae40e066.slice/crio-28db47c3f749ad50c4f85da15abab01138ea22e83439c3bba8015bef2599e911 WatchSource:0}: Error finding container 28db47c3f749ad50c4f85da15abab01138ea22e83439c3bba8015bef2599e911: Status 404 returned error can't find the container with id 28db47c3f749ad50c4f85da15abab01138ea22e83439c3bba8015bef2599e911 Oct 30 16:53:17.254099 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.254070 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:53:17.254185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.254129 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:53:17.254185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.254157 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:53:17.264697 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.264665 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf"] Oct 30 16:53:17.280503 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.280482 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-cb6251-serving-cert\"" Oct 30 16:53:17.318159 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.318115 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:17.318357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.318298 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsn27\" (UniqueName: \"kubernetes.io/projected/f0f77f41-bbe0-4095-8102-f685cc81e7e3-kube-api-access-rsn27\") pod \"maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:17.318473 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.318456 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0f77f41-bbe0-4095-8102-f685cc81e7e3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:17.318681 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.318651 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:17.422565 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.422525 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:17.422883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.422713 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:17.422883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.422797 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsn27\" (UniqueName: \"kubernetes.io/projected/f0f77f41-bbe0-4095-8102-f685cc81e7e3-kube-api-access-rsn27\") pod \"maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:17.422883 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.422873 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0f77f41-bbe0-4095-8102-f685cc81e7e3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:17.423545 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.423521 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0f77f41-bbe0-4095-8102-f685cc81e7e3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:17.423669 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:53:17.423632 4861 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-serving-cert: secret "maas-api-e2e-shared-a-w2-cb6251-serving-cert" not found Oct 30 16:53:17.423771 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:53:17.423694 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-maas-api-tls podName:f0f77f41-bbe0-4095-8102-f685cc81e7e3 nodeName:}" failed. No retries permitted until 2026-10-30 16:53:17.923674038 +0000 UTC m=+1828.266348058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-maas-api-tls") pod "maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" (UID: "f0f77f41-bbe0-4095-8102-f685cc81e7e3") : secret "maas-api-e2e-shared-a-w2-cb6251-serving-cert" not found Oct 30 16:53:17.439691 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.438492 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:17.487808 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.486238 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsn27\" (UniqueName: \"kubernetes.io/projected/f0f77f41-bbe0-4095-8102-f685cc81e7e3-kube-api-access-rsn27\") pod \"maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:17.548793 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.548757 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" event={"ID":"41e4e14f-88e4-4fac-a6fd-04baae40e066","Type":"ContainerStarted","Data":"9db8cf48652149075be5468639cbbb7f230e0b57b594b8ca7ffe649b13560d8c"} Oct 30 16:53:17.548977 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.548812 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" event={"ID":"41e4e14f-88e4-4fac-a6fd-04baae40e066","Type":"ContainerStarted","Data":"28db47c3f749ad50c4f85da15abab01138ea22e83439c3bba8015bef2599e911"} Oct 30 16:53:17.598358 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.598292 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" podStartSLOduration=1.5982722759999999 podStartE2EDuration="1.598272276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:17.594848282 +0000 UTC m=+1827.937522321" watchObservedRunningTime="2026-10-30 16:53:17.598272276 +0000 UTC m=+1827.940946315" Oct 30 16:53:17.898461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.898386 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:17.942923 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.942884 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:17.945534 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:17.945507 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:18.166794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:18.159162 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:18.533192 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:18.516084 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf"] Oct 30 16:53:18.584748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:18.574428 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" event={"ID":"f0f77f41-bbe0-4095-8102-f685cc81e7e3","Type":"ContainerStarted","Data":"5b9a9f633101667108cb31d01843ce4924a73a08ff9e44c8611e5e4452197c6a"} Oct 30 16:53:18.913154 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:18.913072 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:19.590928 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:19.590887 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" event={"ID":"f0f77f41-bbe0-4095-8102-f685cc81e7e3","Type":"ContainerStarted","Data":"2c64a048025213f83cd6d4b8829ba4000f966b6bc82fd288681e1e601c691721"} Oct 30 16:53:19.591144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:19.590983 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:19.591434 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:19.591409 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:19.592495 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:19.592474 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:53:19.634803 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:19.634759 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" podStartSLOduration=2.247545537 podStartE2EDuration="2.634710973s" totalImagesPullingTime="387.165436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:19.634157436 +0000 UTC m=+1829.976831473" watchObservedRunningTime="2026-10-30 16:53:19.634710973 +0000 UTC m=+1829.977385010" Oct 30 16:53:20.290474 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:53:20.290440 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:53:22.125790 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:22.125748 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj"] Oct 30 16:53:22.130220 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:22.130199 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" Oct 30 16:53:22.150681 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:22.150654 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj"] Oct 30 16:53:22.158091 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:22.158067 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-cf4621-dockercfg-7xrh9\"" Oct 30 16:53:22.210748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:22.210060 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-cf4621\"" Oct 30 16:53:22.306751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:22.303339 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lcgm\" (UniqueName: \"kubernetes.io/projected/bb5c8a89-0cb6-4225-97c9-189984852da9-kube-api-access-9lcgm\") pod \"payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj\" (UID: \"bb5c8a89-0cb6-4225-97c9-189984852da9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" Oct 30 16:53:22.370792 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:22.328984 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb5c8a89-0cb6-4225-97c9-189984852da9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj\" (UID: \"bb5c8a89-0cb6-4225-97c9-189984852da9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" Oct 30 16:53:22.430640 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:22.430544 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lcgm\" (UniqueName: \"kubernetes.io/projected/bb5c8a89-0cb6-4225-97c9-189984852da9-kube-api-access-9lcgm\") pod \"payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj\" (UID: \"bb5c8a89-0cb6-4225-97c9-189984852da9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" Oct 30 16:53:22.431248 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:22.431217 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb5c8a89-0cb6-4225-97c9-189984852da9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj\" (UID: \"bb5c8a89-0cb6-4225-97c9-189984852da9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" Oct 30 16:53:22.435979 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:22.435933 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb5c8a89-0cb6-4225-97c9-189984852da9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj\" (UID: \"bb5c8a89-0cb6-4225-97c9-189984852da9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" Oct 30 16:53:22.461150 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:22.461118 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lcgm\" (UniqueName: \"kubernetes.io/projected/bb5c8a89-0cb6-4225-97c9-189984852da9-kube-api-access-9lcgm\") pod \"payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj\" (UID: \"bb5c8a89-0cb6-4225-97c9-189984852da9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" Oct 30 16:53:22.744367 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:22.744333 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" Oct 30 16:53:23.056994 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:23.056965 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj"] Oct 30 16:53:23.058292 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:53:23.058266 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5c8a89_0cb6_4225_97c9_189984852da9.slice/crio-9ab4daa57e3f93a283674b86239cac176e13a8fca29321ed58214096c833a8e1 WatchSource:0}: Error finding container 9ab4daa57e3f93a283674b86239cac176e13a8fca29321ed58214096c833a8e1: Status 404 returned error can't find the container with id 9ab4daa57e3f93a283674b86239cac176e13a8fca29321ed58214096c833a8e1 Oct 30 16:53:23.634666 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:23.634635 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" event={"ID":"bb5c8a89-0cb6-4225-97c9-189984852da9","Type":"ContainerStarted","Data":"cb25c8018e819a603966a3be766e08561e425f90d830120d13ad795ff13280ae"} Oct 30 16:53:23.635116 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:23.634694 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" event={"ID":"bb5c8a89-0cb6-4225-97c9-189984852da9","Type":"ContainerStarted","Data":"9ab4daa57e3f93a283674b86239cac176e13a8fca29321ed58214096c833a8e1"} Oct 30 16:53:23.635116 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:23.634739 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" Oct 30 16:53:25.602979 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:25.602943 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 16:53:25.649976 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:25.649929 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" podStartSLOduration=3.649915718 podStartE2EDuration="3.649915718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:23.704798356 +0000 UTC m=+1834.047472391" watchObservedRunningTime="2026-10-30 16:53:25.649915718 +0000 UTC m=+1835.992589755" Oct 30 16:53:30.330948 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:30.315146 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m"] Oct 30 16:53:30.330948 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:30.315469 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" podUID="963c434d-5cb8-4c34-bc1c-388e3cb1e125" containerName="istio-proxy" containerID="cri-o://68b7249d6114c332b25b1148e3da19b0920dd8cfc835a5c520d0e8e334a943ad" gracePeriod=30 Oct 30 16:53:30.557431 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:30.557344 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592"] Oct 30 16:53:30.581355 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:30.581259 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" podUID="5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" containerName="istio-proxy" containerID="cri-o://f08942ba360fbfa68e374e3eb5c9834484277675a096f25fa24421deb31012db" gracePeriod=30 Oct 30 16:53:30.844744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:30.844617 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd"] Oct 30 16:53:30.845277 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:30.845248 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" podUID="c2cb80d7-4995-4584-8781-0c5921d6115c" containerName="istio-proxy" containerID="cri-o://02f3bff03b8d3ec09a0cd116b78411a940ff1b088487ee61bde855dc8c0786eb" gracePeriod=30 Oct 30 16:53:34.645163 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:34.645133 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" Oct 30 16:53:35.886159 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:35.886133 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:53:35.998612 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:35.998579 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-envoy\") pod \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " Oct 30 16:53:35.998796 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:35.998640 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-token\" (UniqueName: \"kubernetes.io/projected/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-token\") pod \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " Oct 30 16:53:35.998796 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:35.998756 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-data\") pod \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " Oct 30 16:53:35.998909 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:35.998820 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istiod-ca-cert\") pod \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " Oct 30 16:53:35.998909 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:35.998872 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-kube-api-access-g5qzc\" (UniqueName: \"kubernetes.io/projected/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-kube-api-access-g5qzc\") pod \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " Oct 30 16:53:35.998995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:35.998916 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-workload-certs\") pod \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " Oct 30 16:53:35.998995 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:35.998954 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-credential-socket\") pod \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " Oct 30 16:53:35.999083 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:35.999072 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-workload-socket\") pod \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " Oct 30 16:53:35.999126 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:35.999112 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-podinfo\") pod \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\" (UID: \"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7\") " Oct 30 16:53:36.032756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.030414 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-kube-api-access-g5qzc" pod "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" (UID: "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7"). InnerVolumeSpecName "kube-api-access-g5qzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:36.032932 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.032862 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-credential-socket" pod "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" (UID: "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:36.033804 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.033020 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-workload-certs" pod "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" (UID: "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:36.035335 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.035304 4861 generic.go:356] "Generic (PLEG): container finished" podID="c2cb80d7-4995-4584-8781-0c5921d6115c" containerID="02f3bff03b8d3ec09a0cd116b78411a940ff1b088487ee61bde855dc8c0786eb" exitCode=0 Oct 30 16:53:36.035490 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.035468 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" event={"ID":"c2cb80d7-4995-4584-8781-0c5921d6115c","Type":"ContainerDied","Data":"02f3bff03b8d3ec09a0cd116b78411a940ff1b088487ee61bde855dc8c0786eb"} Oct 30 16:53:36.036605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.036343 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-workload-socket" pod "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" (UID: "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:36.036605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.036432 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-data" pod "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" (UID: "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:36.036605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.036575 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istiod-ca-cert" pod "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" (UID: "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:53:36.039509 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.038776 4861 generic.go:356] "Generic (PLEG): container finished" podID="5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" containerID="f08942ba360fbfa68e374e3eb5c9834484277675a096f25fa24421deb31012db" exitCode=0 Oct 30 16:53:36.039509 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.038905 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" event={"ID":"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7","Type":"ContainerDied","Data":"f08942ba360fbfa68e374e3eb5c9834484277675a096f25fa24421deb31012db"} Oct 30 16:53:36.039509 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.038953 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" event={"ID":"5e22d1e7-4404-470d-9bda-3fd5e1a67ed7","Type":"ContainerDied","Data":"7050ec73eb47fef325b8e70a02d68edbaa6aaaef918848c2999ddffbef7915f7"} Oct 30 16:53:36.039509 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.038984 4861 scope.go:122] "RemoveContainer" containerID="f08942ba360fbfa68e374e3eb5c9834484277675a096f25fa24421deb31012db" Oct 30 16:53:36.039509 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.039167 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592" Oct 30 16:53:36.042011 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.041637 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-token" pod "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" (UID: "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:36.042442 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.042125 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-podinfo" pod "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" (UID: "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:53:36.042442 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.042175 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-envoy" pod "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" (UID: "5e22d1e7-4404-470d-9bda-3fd5e1a67ed7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:36.043222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.043066 4861 generic.go:356] "Generic (PLEG): container finished" podID="963c434d-5cb8-4c34-bc1c-388e3cb1e125" containerID="68b7249d6114c332b25b1148e3da19b0920dd8cfc835a5c520d0e8e334a943ad" exitCode=0 Oct 30 16:53:36.043222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.043129 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" event={"ID":"963c434d-5cb8-4c34-bc1c-388e3cb1e125","Type":"ContainerDied","Data":"68b7249d6114c332b25b1148e3da19b0920dd8cfc835a5c520d0e8e334a943ad"} Oct 30 16:53:36.043222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.043157 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" event={"ID":"963c434d-5cb8-4c34-bc1c-388e3cb1e125","Type":"ContainerDied","Data":"195dbd493756818d90f7418095627fac9b3199dbac58dff51e488a149378ad79"} Oct 30 16:53:36.043222 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.043172 4861 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="195dbd493756818d90f7418095627fac9b3199dbac58dff51e488a149378ad79" Oct 30 16:53:36.044009 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.043950 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:53:36.081550 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.080174 4861 scope.go:122] "RemoveContainer" containerID="f08942ba360fbfa68e374e3eb5c9834484277675a096f25fa24421deb31012db" Oct 30 16:53:36.081550 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:53:36.081314 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08942ba360fbfa68e374e3eb5c9834484277675a096f25fa24421deb31012db\": container with ID starting with f08942ba360fbfa68e374e3eb5c9834484277675a096f25fa24421deb31012db not found: ID does not exist" containerID="f08942ba360fbfa68e374e3eb5c9834484277675a096f25fa24421deb31012db" Oct 30 16:53:36.081550 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.081360 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08942ba360fbfa68e374e3eb5c9834484277675a096f25fa24421deb31012db"} err="failed to get container status \"f08942ba360fbfa68e374e3eb5c9834484277675a096f25fa24421deb31012db\": rpc error: code = NotFound desc = could not find container \"f08942ba360fbfa68e374e3eb5c9834484277675a096f25fa24421deb31012db\": container with ID starting with f08942ba360fbfa68e374e3eb5c9834484277675a096f25fa24421deb31012db not found: ID does not exist" Oct 30 16:53:36.081550 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.081388 4861 scope.go:122] "RemoveContainer" containerID="1ec12ff8a0117350f2169865ef2a67e7d6d587e6fe51e906c512605d7cb5c01d" Oct 30 16:53:36.104799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.100766 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-token\" (UniqueName: \"kubernetes.io/projected/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-token\") pod \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " Oct 30 16:53:36.104799 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.100832 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-data\") pod \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " Oct 30 16:53:36.114823 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.114226 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963c434d-5cb8-4c34-bc1c-388e3cb1e125-kube-api-access-zwpxg\" (UniqueName: \"kubernetes.io/projected/963c434d-5cb8-4c34-bc1c-388e3cb1e125-kube-api-access-zwpxg\") pod \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " Oct 30 16:53:36.114823 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.114293 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-credential-socket\") pod \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " Oct 30 16:53:36.114823 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.114380 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-workload-socket\") pod \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " Oct 30 16:53:36.114823 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.114445 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istiod-ca-cert\") pod \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " Oct 30 16:53:36.114823 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.114533 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-workload-certs\") pod \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " Oct 30 16:53:36.114823 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.114563 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-envoy\") pod \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " Oct 30 16:53:36.114823 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.114619 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-podinfo\") pod \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\" (UID: \"963c434d-5cb8-4c34-bc1c-388e3cb1e125\") " Oct 30 16:53:36.139809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.134550 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.139809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.134582 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.139809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.134596 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.139809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.134609 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.139809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.134624 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5qzc\" (UniqueName: \"kubernetes.io/projected/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-kube-api-access-g5qzc\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.139809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.134640 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.139809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.134656 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.139809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.134668 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.139809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.134682 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.139809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.112831 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-data" pod "963c434d-5cb8-4c34-bc1c-388e3cb1e125" (UID: "963c434d-5cb8-4c34-bc1c-388e3cb1e125"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:36.139809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.135344 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-credential-socket" pod "963c434d-5cb8-4c34-bc1c-388e3cb1e125" (UID: "963c434d-5cb8-4c34-bc1c-388e3cb1e125"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:36.139809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.135538 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-workload-certs" pod "963c434d-5cb8-4c34-bc1c-388e3cb1e125" (UID: "963c434d-5cb8-4c34-bc1c-388e3cb1e125"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:36.139809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.135701 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-workload-socket" pod "963c434d-5cb8-4c34-bc1c-388e3cb1e125" (UID: "963c434d-5cb8-4c34-bc1c-388e3cb1e125"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:36.139809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.136123 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-token" pod "963c434d-5cb8-4c34-bc1c-388e3cb1e125" (UID: "963c434d-5cb8-4c34-bc1c-388e3cb1e125"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:36.139809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.136864 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istiod-ca-cert" pod "963c434d-5cb8-4c34-bc1c-388e3cb1e125" (UID: "963c434d-5cb8-4c34-bc1c-388e3cb1e125"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:53:36.139809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.137667 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-podinfo" pod "963c434d-5cb8-4c34-bc1c-388e3cb1e125" (UID: "963c434d-5cb8-4c34-bc1c-388e3cb1e125"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:53:36.140429 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.140352 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-envoy" pod "963c434d-5cb8-4c34-bc1c-388e3cb1e125" (UID: "963c434d-5cb8-4c34-bc1c-388e3cb1e125"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:36.140497 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.140480 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963c434d-5cb8-4c34-bc1c-388e3cb1e125-kube-api-access-zwpxg" pod "963c434d-5cb8-4c34-bc1c-388e3cb1e125" (UID: "963c434d-5cb8-4c34-bc1c-388e3cb1e125"). InnerVolumeSpecName "kube-api-access-zwpxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:36.236185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.236150 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.236185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.236182 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.236185 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.236191 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.236460 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.236199 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.236460 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.236207 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.236460 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.236216 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.236460 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.236225 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwpxg\" (UniqueName: \"kubernetes.io/projected/963c434d-5cb8-4c34-bc1c-388e3cb1e125-kube-api-access-zwpxg\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.236460 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.236234 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.236460 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.236244 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963c434d-5cb8-4c34-bc1c-388e3cb1e125-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.281056 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.281029 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:53:36.396388 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.396295 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592"] Oct 30 16:53:36.400592 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.400569 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-x7592"] Oct 30 16:53:36.437827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.437794 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-credential-socket\") pod \"c2cb80d7-4995-4584-8781-0c5921d6115c\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " Oct 30 16:53:36.438003 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.437856 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-podinfo\") pod \"c2cb80d7-4995-4584-8781-0c5921d6115c\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " Oct 30 16:53:36.438003 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.437948 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-token\" (UniqueName: \"kubernetes.io/projected/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-token\") pod \"c2cb80d7-4995-4584-8781-0c5921d6115c\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " Oct 30 16:53:36.438003 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.437983 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-workload-certs\") pod \"c2cb80d7-4995-4584-8781-0c5921d6115c\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " Oct 30 16:53:36.438189 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.438078 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2cb80d7-4995-4584-8781-0c5921d6115c-kube-api-access-57crd\" (UniqueName: \"kubernetes.io/projected/c2cb80d7-4995-4584-8781-0c5921d6115c-kube-api-access-57crd\") pod \"c2cb80d7-4995-4584-8781-0c5921d6115c\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " Oct 30 16:53:36.438189 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.438103 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-credential-socket" pod "c2cb80d7-4995-4584-8781-0c5921d6115c" (UID: "c2cb80d7-4995-4584-8781-0c5921d6115c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:36.438189 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.438128 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-data\") pod \"c2cb80d7-4995-4584-8781-0c5921d6115c\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " Oct 30 16:53:36.438350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.438241 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2cb80d7-4995-4584-8781-0c5921d6115c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2cb80d7-4995-4584-8781-0c5921d6115c-istiod-ca-cert\") pod \"c2cb80d7-4995-4584-8781-0c5921d6115c\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " Oct 30 16:53:36.438350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.438256 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-workload-certs" pod "c2cb80d7-4995-4584-8781-0c5921d6115c" (UID: "c2cb80d7-4995-4584-8781-0c5921d6115c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:36.438350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.438291 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-envoy\") pod \"c2cb80d7-4995-4584-8781-0c5921d6115c\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " Oct 30 16:53:36.438519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.438428 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-workload-socket\") pod \"c2cb80d7-4995-4584-8781-0c5921d6115c\" (UID: \"c2cb80d7-4995-4584-8781-0c5921d6115c\") " Oct 30 16:53:36.438519 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.438446 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-data" pod "c2cb80d7-4995-4584-8781-0c5921d6115c" (UID: "c2cb80d7-4995-4584-8781-0c5921d6115c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:36.438737 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.438689 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2cb80d7-4995-4584-8781-0c5921d6115c-istiod-ca-cert" pod "c2cb80d7-4995-4584-8781-0c5921d6115c" (UID: "c2cb80d7-4995-4584-8781-0c5921d6115c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:53:36.439046 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.438972 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-workload-socket" pod "c2cb80d7-4995-4584-8781-0c5921d6115c" (UID: "c2cb80d7-4995-4584-8781-0c5921d6115c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:36.439169 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.439048 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.439169 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.439071 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.439169 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.439084 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2cb80d7-4995-4584-8781-0c5921d6115c-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.439169 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.439096 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.440436 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.440409 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-podinfo" pod "c2cb80d7-4995-4584-8781-0c5921d6115c" (UID: "c2cb80d7-4995-4584-8781-0c5921d6115c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:53:36.440817 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.440701 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2cb80d7-4995-4584-8781-0c5921d6115c-kube-api-access-57crd" pod "c2cb80d7-4995-4584-8781-0c5921d6115c" (UID: "c2cb80d7-4995-4584-8781-0c5921d6115c"). InnerVolumeSpecName "kube-api-access-57crd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:36.440996 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.440976 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-token" pod "c2cb80d7-4995-4584-8781-0c5921d6115c" (UID: "c2cb80d7-4995-4584-8781-0c5921d6115c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:53:36.441158 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.441139 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-envoy" pod "c2cb80d7-4995-4584-8781-0c5921d6115c" (UID: "c2cb80d7-4995-4584-8781-0c5921d6115c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:53:36.542777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.540832 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.542777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.540874 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.542777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.540890 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.542777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.540905 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57crd\" (UniqueName: \"kubernetes.io/projected/c2cb80d7-4995-4584-8781-0c5921d6115c-kube-api-access-57crd\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:36.542777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:36.540923 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2cb80d7-4995-4584-8781-0c5921d6115c-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:53:37.108827 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:37.108750 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" Oct 30 16:53:37.109688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:37.108875 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd" event={"ID":"c2cb80d7-4995-4584-8781-0c5921d6115c","Type":"ContainerDied","Data":"29a4f9f614d5ba680c341cada252c5bacc5ae33f4d7f6e5c1c97a671df89be96"} Oct 30 16:53:37.109915 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:37.109752 4861 scope.go:122] "RemoveContainer" containerID="02f3bff03b8d3ec09a0cd116b78411a940ff1b088487ee61bde855dc8c0786eb" Oct 30 16:53:37.115367 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:37.115347 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m" Oct 30 16:53:37.169354 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:37.169328 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd"] Oct 30 16:53:37.182527 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:37.182497 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-cfqjd"] Oct 30 16:53:37.257789 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:37.257757 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m"] Oct 30 16:53:37.262197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:37.262167 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-52f3fd-openshift-default-d4d48dd6f-96g9m"] Oct 30 16:53:37.811346 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:37.811310 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" path="/var/lib/kubelet/pods/5e22d1e7-4404-470d-9bda-3fd5e1a67ed7/volumes" Oct 30 16:53:37.811851 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:37.811834 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963c434d-5cb8-4c34-bc1c-388e3cb1e125" path="/var/lib/kubelet/pods/963c434d-5cb8-4c34-bc1c-388e3cb1e125/volumes" Oct 30 16:53:37.812302 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:37.812287 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2cb80d7-4995-4584-8781-0c5921d6115c" path="/var/lib/kubelet/pods/c2cb80d7-4995-4584-8781-0c5921d6115c/volumes" Oct 30 16:53:41.148469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:41.148436 4861 generic.go:356] "Generic (PLEG): container finished" podID="db2f1614-1fc7-437b-89c8-001c8f180877" containerID="898a23a78e9f5fdabab7bee16d8bbc1ce72c5364ba30b91a0c4a64de0430918c" exitCode=0 Oct 30 16:53:41.148952 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:41.148496 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" event={"ID":"db2f1614-1fc7-437b-89c8-001c8f180877","Type":"ContainerDied","Data":"898a23a78e9f5fdabab7bee16d8bbc1ce72c5364ba30b91a0c4a64de0430918c"} Oct 30 16:53:41.148952 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:41.148900 4861 scope.go:122] "RemoveContainer" containerID="898a23a78e9f5fdabab7bee16d8bbc1ce72c5364ba30b91a0c4a64de0430918c" Oct 30 16:53:41.150354 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:41.150319 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:53:41.150482 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:41.150406 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:53:41.150482 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:41.150454 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:53:42.213749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.209764 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" event={"ID":"db2f1614-1fc7-437b-89c8-001c8f180877","Type":"ContainerStarted","Data":"dfe95bef70a29c1453883c4e11ee85ca50d74f6962b07ac4859707f8cda71ee9"} Oct 30 16:53:42.733769 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.730204 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:53:42.755775 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.740838 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 16:53:42.755775 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.740917 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" podUID="db2f1614-1fc7-437b-89c8-001c8f180877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 16:53:42.797763 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.797699 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8"] Oct 30 16:53:42.798654 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.798629 4861 cpu_manager.go:414] "containerMap: removing container" podUID="963c434d-5cb8-4c34-bc1c-388e3cb1e125" containerName="istio-proxy" Oct 30 16:53:42.798654 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.798648 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963c434d-5cb8-4c34-bc1c-388e3cb1e125" containerName="istio-proxy" Oct 30 16:53:42.798884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.798663 4861 cpu_manager.go:414] "containerMap: removing container" podUID="5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" containerName="istio-proxy" Oct 30 16:53:42.798884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.798673 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" containerName="istio-proxy" Oct 30 16:53:42.798884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.798691 4861 cpu_manager.go:414] "containerMap: removing container" podUID="c2cb80d7-4995-4584-8781-0c5921d6115c" containerName="istio-proxy" Oct 30 16:53:42.798884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.798698 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2cb80d7-4995-4584-8781-0c5921d6115c" containerName="istio-proxy" Oct 30 16:53:42.798884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.798710 4861 cpu_manager.go:414] "containerMap: removing container" podUID="963c434d-5cb8-4c34-bc1c-388e3cb1e125" containerName="istio-proxy" Oct 30 16:53:42.798884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.798716 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963c434d-5cb8-4c34-bc1c-388e3cb1e125" containerName="istio-proxy" Oct 30 16:53:42.799160 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.798958 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="963c434d-5cb8-4c34-bc1c-388e3cb1e125" containerName="istio-proxy" Oct 30 16:53:42.799160 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.798969 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e22d1e7-4404-470d-9bda-3fd5e1a67ed7" containerName="istio-proxy" Oct 30 16:53:42.799160 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.798976 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2cb80d7-4995-4584-8781-0c5921d6115c" containerName="istio-proxy" Oct 30 16:53:42.812938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.812912 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 16:53:42.844835 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.844788 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-cf4621-serving-cert\"" Oct 30 16:53:42.853785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.845045 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8"] Oct 30 16:53:42.876389 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.876344 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8\" (UID: \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 16:53:42.876550 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.876525 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8\" (UID: \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 16:53:42.876637 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.876572 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8\" (UID: \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 16:53:42.877333 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.877301 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnpsv\" (UniqueName: \"kubernetes.io/projected/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-kube-api-access-lnpsv\") pod \"maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8\" (UID: \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 16:53:42.979147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.979113 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnpsv\" (UniqueName: \"kubernetes.io/projected/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-kube-api-access-lnpsv\") pod \"maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8\" (UID: \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 16:53:42.979330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.979176 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8\" (UID: \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 16:53:42.979330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.979257 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8\" (UID: \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 16:53:42.979330 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.979274 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8\" (UID: \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 16:53:42.979988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.979963 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8\" (UID: \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 16:53:42.982030 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.982004 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8\" (UID: \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 16:53:42.982143 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:42.982028 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8\" (UID: \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 16:53:43.031845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:43.025047 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnpsv\" (UniqueName: \"kubernetes.io/projected/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-kube-api-access-lnpsv\") pod \"maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8\" (UID: \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 16:53:43.161473 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:43.161442 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 16:53:43.523747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:43.521775 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8"] Oct 30 16:53:43.525981 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:53:43.525947 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad5ad27_a300_44f3_80af_ac24aabf8ebd.slice/crio-038ac54b49b231429855e10d33c8911d46eea0ae79ff0deac36ac596a6a332c9 WatchSource:0}: Error finding container 038ac54b49b231429855e10d33c8911d46eea0ae79ff0deac36ac596a6a332c9: Status 404 returned error can't find the container with id 038ac54b49b231429855e10d33c8911d46eea0ae79ff0deac36ac596a6a332c9 Oct 30 16:53:43.705538 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:43.705106 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 16:53:43.705538 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:43.705167 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" podUID="db2f1614-1fc7-437b-89c8-001c8f180877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 16:53:44.329856 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:44.318123 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" event={"ID":"1ad5ad27-a300-44f3-80af-ac24aabf8ebd","Type":"ContainerStarted","Data":"0b80fe178f7223ae71baa94622e6726fbb8e46df2b175988f094fb30007d9239"} Oct 30 16:53:44.329856 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:44.318193 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" event={"ID":"1ad5ad27-a300-44f3-80af-ac24aabf8ebd","Type":"ContainerStarted","Data":"038ac54b49b231429855e10d33c8911d46eea0ae79ff0deac36ac596a6a332c9"} Oct 30 16:53:44.329856 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:44.326238 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 16:53:44.372699 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:44.372649 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" podStartSLOduration=1.972308956 podStartE2EDuration="2.372633512s" totalImagesPullingTime="400.324556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:44.37191761 +0000 UTC m=+1854.714591649" watchObservedRunningTime="2026-10-30 16:53:44.372633512 +0000 UTC m=+1854.715307549" Oct 30 16:53:44.716808 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:44.710847 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 16:53:44.716808 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:44.710913 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" podUID="db2f1614-1fc7-437b-89c8-001c8f180877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 16:53:44.955177 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:44.955144 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:53:45.726750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:45.715572 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 16:53:45.726750 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:45.715631 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" podUID="db2f1614-1fc7-437b-89c8-001c8f180877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 16:53:46.149659 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.149056 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp"] Oct 30 16:53:46.154001 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.150785 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="963c434d-5cb8-4c34-bc1c-388e3cb1e125" containerName="istio-proxy" Oct 30 16:53:46.155049 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.155028 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp"] Oct 30 16:53:46.155186 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.155124 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.170844 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.170821 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-07cb3c\"/\"openshift-service-ca.crt\"" Oct 30 16:53:46.183093 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.183068 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-07cb3c\"/\"default-dockercfg-l5qzg\"" Oct 30 16:53:46.183302 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.183287 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-07cb3c\"/\"kube-root-ca.crt\"" Oct 30 16:53:46.183443 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.183429 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-07cb3c\"/\"praxis-route-07cb3c-kserve-self-signed-certs\"" Oct 30 16:53:46.269087 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.269049 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-tmp-dir\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.269265 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.269186 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-home\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.269265 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.269218 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-model-cache\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.269430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.269409 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b17bec-f0b9-492a-b059-fa50daf2ef63-tls-certs\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.269485 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.269471 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-dshm\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.269555 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.269535 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7hcl\" (UniqueName: \"kubernetes.io/projected/b0b17bec-f0b9-492a-b059-fa50daf2ef63-kube-api-access-w7hcl\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.383749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.376620 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b17bec-f0b9-492a-b059-fa50daf2ef63-tls-certs\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.383749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.376739 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-dshm\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.383749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.376784 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7hcl\" (UniqueName: \"kubernetes.io/projected/b0b17bec-f0b9-492a-b059-fa50daf2ef63-kube-api-access-w7hcl\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.383749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.376844 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-tmp-dir\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.383749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.376954 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-home\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.383749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.376981 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-model-cache\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.383749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.378762 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-tmp-dir\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.383749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.379507 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-home\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.383749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.380383 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-model-cache\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.404795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.401414 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-dshm\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.404795 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.402002 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b17bec-f0b9-492a-b059-fa50daf2ef63-tls-certs\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.431789 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.420145 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7hcl\" (UniqueName: \"kubernetes.io/projected/b0b17bec-f0b9-492a-b059-fa50daf2ef63-kube-api-access-w7hcl\") pod \"praxis-route-07cb3c-kserve-545b66c4d5-4r6gp\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.481749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.481060 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:46.727930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.713048 4861 patch_prober.go:29] interesting pod/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 16:53:46.727930 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.713117 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" podUID="db2f1614-1fc7-437b-89c8-001c8f180877" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 16:53:46.857007 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:46.856974 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp"] Oct 30 16:53:46.860980 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:53:46.860935 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0b17bec_f0b9_492a_b059_fa50daf2ef63.slice/crio-efc91bfe7e6027a68f9bc62028e9d43e0f44d6f8af006befc089151ce690923f WatchSource:0}: Error finding container efc91bfe7e6027a68f9bc62028e9d43e0f44d6f8af006befc089151ce690923f: Status 404 returned error can't find the container with id efc91bfe7e6027a68f9bc62028e9d43e0f44d6f8af006befc089151ce690923f Oct 30 16:53:47.407147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:47.407046 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" event={"ID":"b0b17bec-f0b9-492a-b059-fa50daf2ef63","Type":"ContainerStarted","Data":"bbfccc005b2a422998df26f74ac8d08fad444e74921371b23750b6409856d3e0"} Oct 30 16:53:47.407147 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:47.407105 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" event={"ID":"b0b17bec-f0b9-492a-b059-fa50daf2ef63","Type":"ContainerStarted","Data":"efc91bfe7e6027a68f9bc62028e9d43e0f44d6f8af006befc089151ce690923f"} Oct 30 16:53:47.482054 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:47.482020 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:47.720805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:47.719041 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:53:47.725311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:47.725284 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:53:47.777746 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:47.772885 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" podStartSLOduration=1.772863193 podStartE2EDuration="1.772863193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:47.459032619 +0000 UTC m=+1857.801706657" watchObservedRunningTime="2026-10-30 16:53:47.772863193 +0000 UTC m=+1858.115537230" Oct 30 16:53:47.899642 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:47.899603 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t"] Oct 30 16:53:47.908887 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:47.908862 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:47.930056 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:47.930021 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t"] Oct 30 16:53:47.951331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:47.951303 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-deb721-openshift-default-dockercfg-bq5wb\"" Oct 30 16:53:48.062303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.061635 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-envoy\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.062303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.061816 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-workload-certs\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.062303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.061854 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b08299-0d08-470e-ab89-d77ecd29a504-istiod-ca-cert\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.062303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.061884 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-data\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.062303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.061911 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-token\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.062303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.062057 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-podinfo\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.062303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.062097 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-workload-socket\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.062303 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.062228 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54495\" (UniqueName: \"kubernetes.io/projected/f3b08299-0d08-470e-ab89-d77ecd29a504-kube-api-access-54495\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.062858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.062328 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-credential-socket\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.163201 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.163164 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-envoy\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.163382 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.163255 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-workload-certs\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.163382 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.163283 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b08299-0d08-470e-ab89-d77ecd29a504-istiod-ca-cert\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.163382 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.163302 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-data\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.163382 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.163320 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-token\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.163615 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.163455 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-podinfo\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.163615 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.163496 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-workload-socket\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.163716 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.163585 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54495\" (UniqueName: \"kubernetes.io/projected/f3b08299-0d08-470e-ab89-d77ecd29a504-kube-api-access-54495\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.163830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.163707 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-workload-certs\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.163989 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.163963 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-workload-socket\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.164080 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.163711 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-credential-socket\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.164139 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.164071 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-data\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.164139 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.164113 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b08299-0d08-470e-ab89-d77ecd29a504-istiod-ca-cert\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.164246 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.164149 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-credential-socket\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.165840 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.165824 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-envoy\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.166260 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.166236 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-podinfo\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.187839 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.187810 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-token\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.187960 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.187935 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54495\" (UniqueName: \"kubernetes.io/projected/f3b08299-0d08-470e-ab89-d77ecd29a504-kube-api-access-54495\") pod \"e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.239881 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.239847 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:48.584365 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.584322 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t"] Oct 30 16:53:48.584598 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:53:48.584578 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b08299_0d08_470e_ab89_d77ecd29a504.slice/crio-01fb66d9568b23e97b6488a069586b339f71bf1516e203e6f458f9d6372bf1fd WatchSource:0}: Error finding container 01fb66d9568b23e97b6488a069586b339f71bf1516e203e6f458f9d6372bf1fd: Status 404 returned error can't find the container with id 01fb66d9568b23e97b6488a069586b339f71bf1516e203e6f458f9d6372bf1fd Oct 30 16:53:48.587250 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.587218 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:53:48.587321 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.587286 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:53:48.587321 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:48.587314 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:53:49.426300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:49.426236 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" event={"ID":"f3b08299-0d08-470e-ab89-d77ecd29a504","Type":"ContainerStarted","Data":"7970daaa67cb227919c0fb172505a6e59d1b591a28a24ed13a174e95e157ac4a"} Oct 30 16:53:49.426822 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:49.426320 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" event={"ID":"f3b08299-0d08-470e-ab89-d77ecd29a504","Type":"ContainerStarted","Data":"01fb66d9568b23e97b6488a069586b339f71bf1516e203e6f458f9d6372bf1fd"} Oct 30 16:53:49.476227 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:49.476166 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" podStartSLOduration=2.47615214 podStartE2EDuration="2.47615214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:49.473226096 +0000 UTC m=+1859.815900147" watchObservedRunningTime="2026-10-30 16:53:49.47615214 +0000 UTC m=+1859.818826237" Oct 30 16:53:50.240708 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:50.240670 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:50.246535 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:50.246510 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:50.291868 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:53:50.291834 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:53:50.433908 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:50.433880 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:50.435006 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:50.434974 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:53:51.359910 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:51.359832 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 16:53:53.881712 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:53.881677 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc"] Oct 30 16:53:53.887314 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:53.887295 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" Oct 30 16:53:53.905247 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:53.905224 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-deb721-dockercfg-2hr8m\"" Oct 30 16:53:53.905851 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:53.905826 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-deb721\"" Oct 30 16:53:53.910547 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:53.910526 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc"] Oct 30 16:53:54.113757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.113159 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77sbf\" (UniqueName: \"kubernetes.io/projected/4b8f2426-9e85-415e-9077-cf2471ad0875-kube-api-access-77sbf\") pod \"payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc\" (UID: \"4b8f2426-9e85-415e-9077-cf2471ad0875\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" Oct 30 16:53:54.113757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.113298 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8f2426-9e85-415e-9077-cf2471ad0875-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc\" (UID: \"4b8f2426-9e85-415e-9077-cf2471ad0875\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" Oct 30 16:53:54.167209 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.167092 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp"] Oct 30 16:53:54.177694 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.177663 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp"] Oct 30 16:53:54.178061 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.178040 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" Oct 30 16:53:54.214861 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.214833 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77sbf\" (UniqueName: \"kubernetes.io/projected/4b8f2426-9e85-415e-9077-cf2471ad0875-kube-api-access-77sbf\") pod \"payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc\" (UID: \"4b8f2426-9e85-415e-9077-cf2471ad0875\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" Oct 30 16:53:54.215057 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.215019 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8f2426-9e85-415e-9077-cf2471ad0875-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc\" (UID: \"4b8f2426-9e85-415e-9077-cf2471ad0875\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" Oct 30 16:53:54.215144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.215098 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md6df\" (UniqueName: \"kubernetes.io/projected/59476740-3ac5-4865-a330-505ee1c8cc0a-kube-api-access-md6df\") pod \"payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp\" (UID: \"59476740-3ac5-4865-a330-505ee1c8cc0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" Oct 30 16:53:54.215293 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.215277 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59476740-3ac5-4865-a330-505ee1c8cc0a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp\" (UID: \"59476740-3ac5-4865-a330-505ee1c8cc0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" Oct 30 16:53:54.215871 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.215852 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8f2426-9e85-415e-9077-cf2471ad0875-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc\" (UID: \"4b8f2426-9e85-415e-9077-cf2471ad0875\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" Oct 30 16:53:54.239099 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.239067 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77sbf\" (UniqueName: \"kubernetes.io/projected/4b8f2426-9e85-415e-9077-cf2471ad0875-kube-api-access-77sbf\") pod \"payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc\" (UID: \"4b8f2426-9e85-415e-9077-cf2471ad0875\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" Oct 30 16:53:54.316591 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.316555 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md6df\" (UniqueName: \"kubernetes.io/projected/59476740-3ac5-4865-a330-505ee1c8cc0a-kube-api-access-md6df\") pod \"payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp\" (UID: \"59476740-3ac5-4865-a330-505ee1c8cc0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" Oct 30 16:53:54.316758 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.316678 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59476740-3ac5-4865-a330-505ee1c8cc0a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp\" (UID: \"59476740-3ac5-4865-a330-505ee1c8cc0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" Oct 30 16:53:54.317406 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.317380 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59476740-3ac5-4865-a330-505ee1c8cc0a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp\" (UID: \"59476740-3ac5-4865-a330-505ee1c8cc0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" Oct 30 16:53:54.341329 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.341295 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md6df\" (UniqueName: \"kubernetes.io/projected/59476740-3ac5-4865-a330-505ee1c8cc0a-kube-api-access-md6df\") pod \"payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp\" (UID: \"59476740-3ac5-4865-a330-505ee1c8cc0a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" Oct 30 16:53:54.501018 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.500989 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" Oct 30 16:53:54.503274 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.503249 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" Oct 30 16:53:54.946845 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:54.943250 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc"] Oct 30 16:53:55.081520 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:55.081494 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp"] Oct 30 16:53:55.082630 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:53:55.082602 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59476740_3ac5_4865_a330_505ee1c8cc0a.slice/crio-cdcb9f68c33b7cb85e880618eb6c4daf77a850eeb428a643773fb65f2b9c9ebb WatchSource:0}: Error finding container cdcb9f68c33b7cb85e880618eb6c4daf77a850eeb428a643773fb65f2b9c9ebb: Status 404 returned error can't find the container with id cdcb9f68c33b7cb85e880618eb6c4daf77a850eeb428a643773fb65f2b9c9ebb Oct 30 16:53:55.540093 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:55.540051 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" event={"ID":"4b8f2426-9e85-415e-9077-cf2471ad0875","Type":"ContainerStarted","Data":"8a8690e3a5cbc1b293e967da4c1c07dae431e98eb0b1aa0078f9da4587de2f24"} Oct 30 16:53:55.540304 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:55.540119 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" event={"ID":"4b8f2426-9e85-415e-9077-cf2471ad0875","Type":"ContainerStarted","Data":"9c924c8e17c917c397cb87976a9650b21804bb94862604607bcca111f0372b15"} Oct 30 16:53:55.540304 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:55.540144 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" Oct 30 16:53:55.542008 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:55.541978 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" event={"ID":"59476740-3ac5-4865-a330-505ee1c8cc0a","Type":"ContainerStarted","Data":"ef438c97c4f6858a81555210e3e8d03606f2e9bbdd7b8d827e9b75a1e8c4ad7c"} Oct 30 16:53:55.542114 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:55.542039 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" event={"ID":"59476740-3ac5-4865-a330-505ee1c8cc0a","Type":"ContainerStarted","Data":"cdcb9f68c33b7cb85e880618eb6c4daf77a850eeb428a643773fb65f2b9c9ebb"} Oct 30 16:53:55.542114 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:55.542077 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" Oct 30 16:53:55.585479 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:55.585431 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" podStartSLOduration=2.585418785 podStartE2EDuration="2.585418785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:55.582539224 +0000 UTC m=+1865.925213331" watchObservedRunningTime="2026-10-30 16:53:55.585418785 +0000 UTC m=+1865.928092822" Oct 30 16:53:55.628906 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:55.628789 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" podStartSLOduration=2.628702242 podStartE2EDuration="2.628702242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:55.625183652 +0000 UTC m=+1865.967857691" watchObservedRunningTime="2026-10-30 16:53:55.628702242 +0000 UTC m=+1865.971376282" Oct 30 16:53:56.482017 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:56.481982 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:56.494787 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:56.494737 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:53:56.561780 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:53:56.561748 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:54:05.281443 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:05.278236 4861 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": read tcp 10.132.0.2:54406->10.132.0.184:15021: read: connection reset by peer" start-of-body= Oct 30 16:54:05.281443 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:05.278291 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" podUID="41e4e14f-88e4-4fac-a6fd-04baae40e066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": read tcp 10.132.0.2:54406->10.132.0.184:15021: read: connection reset by peer" Oct 30 16:54:05.718128 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:05.718095 4861 generic.go:356] "Generic (PLEG): container finished" podID="41e4e14f-88e4-4fac-a6fd-04baae40e066" containerID="9db8cf48652149075be5468639cbbb7f230e0b57b594b8ca7ffe649b13560d8c" exitCode=0 Oct 30 16:54:05.718324 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:05.718153 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" event={"ID":"41e4e14f-88e4-4fac-a6fd-04baae40e066","Type":"ContainerDied","Data":"9db8cf48652149075be5468639cbbb7f230e0b57b594b8ca7ffe649b13560d8c"} Oct 30 16:54:05.718581 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:05.718563 4861 scope.go:122] "RemoveContainer" containerID="9db8cf48652149075be5468639cbbb7f230e0b57b594b8ca7ffe649b13560d8c" Oct 30 16:54:05.720220 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:05.720194 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:54:05.720294 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:05.720258 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:54:05.720348 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:05.720291 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:54:06.554076 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:06.554047 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" Oct 30 16:54:06.554513 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:06.554130 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" Oct 30 16:54:06.745413 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:06.745369 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" event={"ID":"41e4e14f-88e4-4fac-a6fd-04baae40e066","Type":"ContainerStarted","Data":"17a2de0fa38cdd7949ff8caa121b3f3823de4b5db372f05d9b95d629d26d5db5"} Oct 30 16:54:06.898083 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:06.897993 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:54:06.898278 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:06.898188 4861 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 16:54:06.898278 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:06.898243 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" podUID="41e4e14f-88e4-4fac-a6fd-04baae40e066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 16:54:07.898676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:07.898643 4861 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 16:54:07.899131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:07.898705 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" podUID="41e4e14f-88e4-4fac-a6fd-04baae40e066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 16:54:08.898198 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:08.898160 4861 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 16:54:08.898373 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:08.898235 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" podUID="41e4e14f-88e4-4fac-a6fd-04baae40e066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 16:54:09.668925 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:09.668895 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb"] Oct 30 16:54:09.670004 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:09.669972 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" podUID="0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" containerName="istio-proxy" containerID="cri-o://e06afb8db231036a8e207efeb3eceafaca53c5c63f3db41b9e9fad53da81d9f1" gracePeriod=30 Oct 30 16:54:09.919802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:09.919426 4861 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 16:54:09.919802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:09.919511 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" podUID="41e4e14f-88e4-4fac-a6fd-04baae40e066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 16:54:10.915747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:10.907140 4861 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 16:54:10.915747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:10.907202 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" podUID="41e4e14f-88e4-4fac-a6fd-04baae40e066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 16:54:11.900137 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:11.900092 4861 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 30 16:54:11.900430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:11.900397 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" podUID="41e4e14f-88e4-4fac-a6fd-04baae40e066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 30 16:54:12.902564 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:12.902530 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:54:12.903021 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:12.902995 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:54:12.903694 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:12.903675 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 16:54:14.819140 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:14.819101 4861 generic.go:356] "Generic (PLEG): container finished" podID="0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" containerID="e06afb8db231036a8e207efeb3eceafaca53c5c63f3db41b9e9fad53da81d9f1" exitCode=0 Oct 30 16:54:14.819602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:14.819506 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" event={"ID":"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d","Type":"ContainerDied","Data":"e06afb8db231036a8e207efeb3eceafaca53c5c63f3db41b9e9fad53da81d9f1"} Oct 30 16:54:14.819602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:14.819592 4861 scope.go:122] "RemoveContainer" containerID="ad95239a93ef7038f555175ae6f5c2f2a6057ff7cc68af02ca184706f06db97e" Oct 30 16:54:14.966291 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:14.966266 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:54:15.055033 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.054991 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-workload-socket\") pod \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " Oct 30 16:54:15.055197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.055059 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-credential-socket\") pod \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " Oct 30 16:54:15.055197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.055108 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-data\") pod \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " Oct 30 16:54:15.055197 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.055137 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-token\" (UniqueName: \"kubernetes.io/projected/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-token\") pod \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " Oct 30 16:54:15.055375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.055242 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-workload-certs\") pod \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " Oct 30 16:54:15.055375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.055303 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-workload-socket" pod "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" (UID: "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:54:15.055375 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.055346 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-kube-api-access-xk56w\" (UniqueName: \"kubernetes.io/projected/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-kube-api-access-xk56w\") pod \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " Oct 30 16:54:15.055546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.055383 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-podinfo\") pod \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " Oct 30 16:54:15.055546 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.055420 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-credential-socket" pod "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" (UID: "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:54:15.055652 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.055555 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-envoy\") pod \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " Oct 30 16:54:15.055652 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.055596 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istiod-ca-cert\") pod \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\" (UID: \"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d\") " Oct 30 16:54:15.055652 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.055605 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-workload-certs" pod "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" (UID: "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:54:15.056085 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.056062 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-data" pod "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" (UID: "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:54:15.056351 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.056196 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:54:15.056545 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.056529 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:54:15.056660 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.056646 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:54:15.056767 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.056337 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istiod-ca-cert" pod "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" (UID: "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:54:15.058187 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.058153 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-token" pod "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" (UID: "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:54:15.058455 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.058417 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-kube-api-access-xk56w" pod "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" (UID: "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d"). InnerVolumeSpecName "kube-api-access-xk56w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:54:15.058774 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.058711 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-podinfo" pod "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" (UID: "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:54:15.058917 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.058897 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-envoy" pod "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" (UID: "0da9ff11-4823-4eec-bfcb-d10ec3e0c83d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:54:15.158461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.158430 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk56w\" (UniqueName: \"kubernetes.io/projected/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-kube-api-access-xk56w\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:54:15.158461 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.158460 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:54:15.158652 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.158472 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:54:15.158652 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.158484 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:54:15.158652 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.158496 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:54:15.158652 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.158506 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:54:15.826706 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.826684 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" Oct 30 16:54:15.827173 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.826714 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb" event={"ID":"0da9ff11-4823-4eec-bfcb-d10ec3e0c83d","Type":"ContainerDied","Data":"89352e9dde61baa244a01f8205dcab0ed332555f0841913e167ffbde8bf16210"} Oct 30 16:54:15.827173 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.826799 4861 scope.go:122] "RemoveContainer" containerID="e06afb8db231036a8e207efeb3eceafaca53c5c63f3db41b9e9fad53da81d9f1" Oct 30 16:54:15.937836 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.937795 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb"] Oct 30 16:54:15.990573 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:15.990532 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-4a8ab1-openshift-default-599bdb57b9-lgwkb"] Oct 30 16:54:17.774052 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.774022 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts"] Oct 30 16:54:17.774667 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.774650 4861 cpu_manager.go:414] "containerMap: removing container" podUID="0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" containerName="istio-proxy" Oct 30 16:54:17.774718 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.774668 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" containerName="istio-proxy" Oct 30 16:54:17.774825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.774816 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" containerName="istio-proxy" Oct 30 16:54:17.779805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.779778 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" Oct 30 16:54:17.807570 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.807535 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-3ca339\"" Oct 30 16:54:17.807988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.807586 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-3ca339-dockercfg-hbkk9\"" Oct 30 16:54:17.813487 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.813458 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" path="/var/lib/kubelet/pods/0da9ff11-4823-4eec-bfcb-d10ec3e0c83d/volumes" Oct 30 16:54:17.814054 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.814014 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts"] Oct 30 16:54:17.848685 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.848659 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq"] Oct 30 16:54:17.849651 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.849630 4861 cpu_manager.go:414] "containerMap: removing container" podUID="0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" containerName="istio-proxy" Oct 30 16:54:17.849651 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.849651 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" containerName="istio-proxy" Oct 30 16:54:17.849837 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.849778 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da9ff11-4823-4eec-bfcb-d10ec3e0c83d" containerName="istio-proxy" Oct 30 16:54:17.853584 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.853563 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" Oct 30 16:54:17.866003 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.865981 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq"] Oct 30 16:54:17.931816 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.927077 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr97j\" (UniqueName: \"kubernetes.io/projected/3b0f1529-cc32-4603-987f-55b7d3a54c2a-kube-api-access-rr97j\") pod \"payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts\" (UID: \"3b0f1529-cc32-4603-987f-55b7d3a54c2a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" Oct 30 16:54:17.931816 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.927141 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199a5f3f-241a-4d65-813c-928af105ac2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq\" (UID: \"199a5f3f-241a-4d65-813c-928af105ac2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" Oct 30 16:54:17.931816 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.927198 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmhzq\" (UniqueName: \"kubernetes.io/projected/199a5f3f-241a-4d65-813c-928af105ac2b-kube-api-access-wmhzq\") pod \"payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq\" (UID: \"199a5f3f-241a-4d65-813c-928af105ac2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" Oct 30 16:54:17.931816 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:17.927261 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0f1529-cc32-4603-987f-55b7d3a54c2a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts\" (UID: \"3b0f1529-cc32-4603-987f-55b7d3a54c2a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" Oct 30 16:54:18.091808 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.091661 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr97j\" (UniqueName: \"kubernetes.io/projected/3b0f1529-cc32-4603-987f-55b7d3a54c2a-kube-api-access-rr97j\") pod \"payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts\" (UID: \"3b0f1529-cc32-4603-987f-55b7d3a54c2a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" Oct 30 16:54:18.091986 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.091749 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199a5f3f-241a-4d65-813c-928af105ac2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq\" (UID: \"199a5f3f-241a-4d65-813c-928af105ac2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" Oct 30 16:54:18.091986 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.091956 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmhzq\" (UniqueName: \"kubernetes.io/projected/199a5f3f-241a-4d65-813c-928af105ac2b-kube-api-access-wmhzq\") pod \"payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq\" (UID: \"199a5f3f-241a-4d65-813c-928af105ac2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" Oct 30 16:54:18.092132 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.092013 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0f1529-cc32-4603-987f-55b7d3a54c2a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts\" (UID: \"3b0f1529-cc32-4603-987f-55b7d3a54c2a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" Oct 30 16:54:18.126009 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.093984 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0f1529-cc32-4603-987f-55b7d3a54c2a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts\" (UID: \"3b0f1529-cc32-4603-987f-55b7d3a54c2a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" Oct 30 16:54:18.127167 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.112936 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199a5f3f-241a-4d65-813c-928af105ac2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq\" (UID: \"199a5f3f-241a-4d65-813c-928af105ac2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" Oct 30 16:54:18.136238 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.136214 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr97j\" (UniqueName: \"kubernetes.io/projected/3b0f1529-cc32-4603-987f-55b7d3a54c2a-kube-api-access-rr97j\") pod \"payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts\" (UID: \"3b0f1529-cc32-4603-987f-55b7d3a54c2a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" Oct 30 16:54:18.136355 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.136218 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmhzq\" (UniqueName: \"kubernetes.io/projected/199a5f3f-241a-4d65-813c-928af105ac2b-kube-api-access-wmhzq\") pod \"payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq\" (UID: \"199a5f3f-241a-4d65-813c-928af105ac2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" Oct 30 16:54:18.170680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.170654 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" Oct 30 16:54:18.394959 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.394873 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" Oct 30 16:54:18.487228 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.487200 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq"] Oct 30 16:54:18.489571 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:54:18.488842 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199a5f3f_241a_4d65_813c_928af105ac2b.slice/crio-e425bef0a39266ecd524cfb11821306efa18ba23454af84a473837c3ff80c930 WatchSource:0}: Error finding container e425bef0a39266ecd524cfb11821306efa18ba23454af84a473837c3ff80c930: Status 404 returned error can't find the container with id e425bef0a39266ecd524cfb11821306efa18ba23454af84a473837c3ff80c930 Oct 30 16:54:18.702202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.702172 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts"] Oct 30 16:54:18.703035 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:54:18.703007 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0f1529_cc32_4603_987f_55b7d3a54c2a.slice/crio-1da294cedef5b7e64027bceaf564e676d0a9831096f0d23fce43e3d0a5f387ae WatchSource:0}: Error finding container 1da294cedef5b7e64027bceaf564e676d0a9831096f0d23fce43e3d0a5f387ae: Status 404 returned error can't find the container with id 1da294cedef5b7e64027bceaf564e676d0a9831096f0d23fce43e3d0a5f387ae Oct 30 16:54:18.854401 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.854368 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" event={"ID":"199a5f3f-241a-4d65-813c-928af105ac2b","Type":"ContainerStarted","Data":"959e2dee1c995a8b124138f3c469ca9529c0f40e745f2e95596293b224fa48c2"} Oct 30 16:54:18.854831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.854427 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" event={"ID":"199a5f3f-241a-4d65-813c-928af105ac2b","Type":"ContainerStarted","Data":"e425bef0a39266ecd524cfb11821306efa18ba23454af84a473837c3ff80c930"} Oct 30 16:54:18.854831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.854457 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" Oct 30 16:54:18.856110 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.856085 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" event={"ID":"3b0f1529-cc32-4603-987f-55b7d3a54c2a","Type":"ContainerStarted","Data":"991db04ba391e2a05aa5a48536271ebebc7b731dbf3f07849783c71eb49dff6f"} Oct 30 16:54:18.856206 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.856132 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" event={"ID":"3b0f1529-cc32-4603-987f-55b7d3a54c2a","Type":"ContainerStarted","Data":"1da294cedef5b7e64027bceaf564e676d0a9831096f0d23fce43e3d0a5f387ae"} Oct 30 16:54:18.856206 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.856189 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" Oct 30 16:54:18.900109 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.900070 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" podStartSLOduration=1.900057592 podStartE2EDuration="1.900057592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:18.895706134 +0000 UTC m=+1889.238380173" watchObservedRunningTime="2026-10-30 16:54:18.900057592 +0000 UTC m=+1889.242731630" Oct 30 16:54:18.947218 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:18.947150 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" podStartSLOduration=1.9471272069999999 podStartE2EDuration="1.947127207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:18.944527738 +0000 UTC m=+1889.287201778" watchObservedRunningTime="2026-10-30 16:54:18.947127207 +0000 UTC m=+1889.289801248" Oct 30 16:54:20.290379 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:54:20.290343 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:54:20.739740 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:20.739701 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6"] Oct 30 16:54:20.745385 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:20.745366 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 16:54:20.769391 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:20.769365 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6"] Oct 30 16:54:20.776407 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:20.776370 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-8bff6e-serving-cert\"" Oct 30 16:54:20.841133 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:20.841091 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs7h8\" (UniqueName: \"kubernetes.io/projected/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-kube-api-access-hs7h8\") pod \"maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6\" (UID: \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 16:54:20.841133 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:20.841136 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6\" (UID: \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 16:54:20.841497 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:20.841480 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6\" (UID: \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 16:54:20.841549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:20.841520 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6\" (UID: \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 16:54:20.950749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:20.944610 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6\" (UID: \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 16:54:20.950749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:20.944704 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs7h8\" (UniqueName: \"kubernetes.io/projected/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-kube-api-access-hs7h8\") pod \"maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6\" (UID: \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 16:54:20.950749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:20.944750 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6\" (UID: \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 16:54:20.950749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:20.945047 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6\" (UID: \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 16:54:20.950749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:20.945937 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6\" (UID: \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 16:54:20.979757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:20.976007 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6\" (UID: \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 16:54:21.005794 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:21.002134 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6\" (UID: \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 16:54:21.010465 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:21.006173 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs7h8\" (UniqueName: \"kubernetes.io/projected/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-kube-api-access-hs7h8\") pod \"maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6\" (UID: \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 16:54:21.060391 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:21.060361 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 16:54:21.375937 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:21.375910 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6"] Oct 30 16:54:21.376436 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:54:21.376412 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ba73dc_2ca3_4c9d_a839_f0a7bd9ec1d6.slice/crio-aaaa15acef7c857e313baf8c07cd177de66e7b6b83962ed565ccbe9853968ec8 WatchSource:0}: Error finding container aaaa15acef7c857e313baf8c07cd177de66e7b6b83962ed565ccbe9853968ec8: Status 404 returned error can't find the container with id aaaa15acef7c857e313baf8c07cd177de66e7b6b83962ed565ccbe9853968ec8 Oct 30 16:54:21.879088 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:21.879054 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" event={"ID":"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6","Type":"ContainerStarted","Data":"d994cc3cd3ec4becd14ce3079ab82e125b864d44b038c36cd46b73dba877d084"} Oct 30 16:54:21.879290 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:21.879112 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" event={"ID":"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6","Type":"ContainerStarted","Data":"aaaa15acef7c857e313baf8c07cd177de66e7b6b83962ed565ccbe9853968ec8"} Oct 30 16:54:21.879290 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:21.879137 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 16:54:21.985753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:21.984959 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" podStartSLOduration=1.649449842 podStartE2EDuration="1.984934016s" totalImagesPullingTime="335.484174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:21.935965032 +0000 UTC m=+1892.278639072" watchObservedRunningTime="2026-10-30 16:54:21.984934016 +0000 UTC m=+1892.327608054" Oct 30 16:54:24.879992 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:24.879958 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz"] Oct 30 16:54:24.884940 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:24.884917 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" Oct 30 16:54:24.901432 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:24.901407 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-799c76\"" Oct 30 16:54:24.901981 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:24.901955 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3272339a-14a7-4af5-90cd-8beb09ec9e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz\" (UID: \"3272339a-14a7-4af5-90cd-8beb09ec9e40\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" Oct 30 16:54:24.902074 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:24.902010 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dk2m\" (UniqueName: \"kubernetes.io/projected/3272339a-14a7-4af5-90cd-8beb09ec9e40-kube-api-access-5dk2m\") pod \"payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz\" (UID: \"3272339a-14a7-4af5-90cd-8beb09ec9e40\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" Oct 30 16:54:24.918765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:24.914097 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-799c76-dockercfg-rlz6z\"" Oct 30 16:54:24.921094 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:24.921067 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz"] Oct 30 16:54:25.087748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.032835 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3272339a-14a7-4af5-90cd-8beb09ec9e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz\" (UID: \"3272339a-14a7-4af5-90cd-8beb09ec9e40\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" Oct 30 16:54:25.087748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.032898 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dk2m\" (UniqueName: \"kubernetes.io/projected/3272339a-14a7-4af5-90cd-8beb09ec9e40-kube-api-access-5dk2m\") pod \"payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz\" (UID: \"3272339a-14a7-4af5-90cd-8beb09ec9e40\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" Oct 30 16:54:25.087748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.049800 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3272339a-14a7-4af5-90cd-8beb09ec9e40-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz\" (UID: \"3272339a-14a7-4af5-90cd-8beb09ec9e40\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" Oct 30 16:54:25.117747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.091589 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dk2m\" (UniqueName: \"kubernetes.io/projected/3272339a-14a7-4af5-90cd-8beb09ec9e40-kube-api-access-5dk2m\") pod \"payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz\" (UID: \"3272339a-14a7-4af5-90cd-8beb09ec9e40\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" Oct 30 16:54:25.160782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.158196 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5"] Oct 30 16:54:25.180939 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.167583 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" Oct 30 16:54:25.180939 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.170438 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5"] Oct 30 16:54:25.201698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.201592 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" Oct 30 16:54:25.257395 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.257354 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g85xh\" (UniqueName: \"kubernetes.io/projected/2f716c0d-9107-46e1-ae3c-7518ad0a62bb-kube-api-access-g85xh\") pod \"payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5\" (UID: \"2f716c0d-9107-46e1-ae3c-7518ad0a62bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" Oct 30 16:54:25.257550 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.257513 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f716c0d-9107-46e1-ae3c-7518ad0a62bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5\" (UID: \"2f716c0d-9107-46e1-ae3c-7518ad0a62bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" Oct 30 16:54:25.358508 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.358478 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f716c0d-9107-46e1-ae3c-7518ad0a62bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5\" (UID: \"2f716c0d-9107-46e1-ae3c-7518ad0a62bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" Oct 30 16:54:25.358698 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.358655 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g85xh\" (UniqueName: \"kubernetes.io/projected/2f716c0d-9107-46e1-ae3c-7518ad0a62bb-kube-api-access-g85xh\") pod \"payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5\" (UID: \"2f716c0d-9107-46e1-ae3c-7518ad0a62bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" Oct 30 16:54:25.359238 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.359217 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f716c0d-9107-46e1-ae3c-7518ad0a62bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5\" (UID: \"2f716c0d-9107-46e1-ae3c-7518ad0a62bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" Oct 30 16:54:25.382642 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.382613 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g85xh\" (UniqueName: \"kubernetes.io/projected/2f716c0d-9107-46e1-ae3c-7518ad0a62bb-kube-api-access-g85xh\") pod \"payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5\" (UID: \"2f716c0d-9107-46e1-ae3c-7518ad0a62bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" Oct 30 16:54:25.503407 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.503382 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" Oct 30 16:54:25.516391 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.516370 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz"] Oct 30 16:54:25.518467 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:54:25.518439 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3272339a_14a7_4af5_90cd_8beb09ec9e40.slice/crio-fa3e0b01b22b189ca234501a83ebd0c1fb5295c11438febd1ff8771bc3b75be8 WatchSource:0}: Error finding container fa3e0b01b22b189ca234501a83ebd0c1fb5295c11438febd1ff8771bc3b75be8: Status 404 returned error can't find the container with id fa3e0b01b22b189ca234501a83ebd0c1fb5295c11438febd1ff8771bc3b75be8 Oct 30 16:54:25.822663 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.822631 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5"] Oct 30 16:54:25.823500 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:54:25.823472 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f716c0d_9107_46e1_ae3c_7518ad0a62bb.slice/crio-addfec97b617bb68d71df665a0f9f7936e43c4a4e30414847a10eb4d312307ac WatchSource:0}: Error finding container addfec97b617bb68d71df665a0f9f7936e43c4a4e30414847a10eb4d312307ac: Status 404 returned error can't find the container with id addfec97b617bb68d71df665a0f9f7936e43c4a4e30414847a10eb4d312307ac Oct 30 16:54:25.908789 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.908758 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" event={"ID":"2f716c0d-9107-46e1-ae3c-7518ad0a62bb","Type":"ContainerStarted","Data":"c66360cd5dd7bff6b9f76eb9e9ba5448f836da652d9b72b11f4c7d1651553ad1"} Oct 30 16:54:25.909202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.908847 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" event={"ID":"2f716c0d-9107-46e1-ae3c-7518ad0a62bb","Type":"ContainerStarted","Data":"addfec97b617bb68d71df665a0f9f7936e43c4a4e30414847a10eb4d312307ac"} Oct 30 16:54:25.909202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.908881 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" Oct 30 16:54:25.910577 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.910548 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" event={"ID":"3272339a-14a7-4af5-90cd-8beb09ec9e40","Type":"ContainerStarted","Data":"1c5f807f6062045272012ec2a972404c00f339412ea27dc62d33f23020fea13a"} Oct 30 16:54:25.910714 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.910594 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" event={"ID":"3272339a-14a7-4af5-90cd-8beb09ec9e40","Type":"ContainerStarted","Data":"fa3e0b01b22b189ca234501a83ebd0c1fb5295c11438febd1ff8771bc3b75be8"} Oct 30 16:54:25.910714 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.910619 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" Oct 30 16:54:25.952513 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.952466 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" podStartSLOduration=1.952452392 podStartE2EDuration="1.952452392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:25.950084426 +0000 UTC m=+1896.292758465" watchObservedRunningTime="2026-10-30 16:54:25.952452392 +0000 UTC m=+1896.295126463" Oct 30 16:54:25.995926 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:25.995882 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" podStartSLOduration=1.9958689920000001 podStartE2EDuration="1.995868992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:25.993972162 +0000 UTC m=+1896.336646201" watchObservedRunningTime="2026-10-30 16:54:25.995868992 +0000 UTC m=+1896.338543030" Oct 30 16:54:27.891669 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:27.891639 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 16:54:29.867095 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:29.867067 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" Oct 30 16:54:29.867529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:29.867344 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" Oct 30 16:54:36.928807 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:36.928776 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" Oct 30 16:54:36.929239 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:36.928910 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" Oct 30 16:54:43.057064 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:43.057011 4861 generic.go:356] "Generic (PLEG): container finished" podID="f3b08299-0d08-470e-ab89-d77ecd29a504" containerID="7970daaa67cb227919c0fb172505a6e59d1b591a28a24ed13a174e95e157ac4a" exitCode=0 Oct 30 16:54:43.057792 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:43.057752 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" event={"ID":"f3b08299-0d08-470e-ab89-d77ecd29a504","Type":"ContainerDied","Data":"7970daaa67cb227919c0fb172505a6e59d1b591a28a24ed13a174e95e157ac4a"} Oct 30 16:54:43.058500 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:43.058472 4861 scope.go:122] "RemoveContainer" containerID="7970daaa67cb227919c0fb172505a6e59d1b591a28a24ed13a174e95e157ac4a" Oct 30 16:54:43.060550 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:43.060521 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:54:43.060650 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:43.060593 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:54:43.060650 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:43.060627 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:54:44.067286 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:44.067253 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" event={"ID":"f3b08299-0d08-470e-ab89-d77ecd29a504","Type":"ContainerStarted","Data":"65360dfe3632012aff1b3847392512cf16af52f5222884b07ffa5e88f0aab0c9"} Oct 30 16:54:44.240380 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:44.240341 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:54:44.240591 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:44.240562 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 16:54:44.240668 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:44.240629 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" podUID="f3b08299-0d08-470e-ab89-d77ecd29a504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 16:54:45.250752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:45.246219 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 16:54:45.250752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:45.246311 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" podUID="f3b08299-0d08-470e-ab89-d77ecd29a504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 16:54:46.240239 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:46.240201 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 16:54:46.240506 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:46.240281 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" podUID="f3b08299-0d08-470e-ab89-d77ecd29a504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 16:54:47.241114 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:47.241080 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 16:54:47.241496 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:47.241140 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" podUID="f3b08299-0d08-470e-ab89-d77ecd29a504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 16:54:48.240311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:48.240277 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 16:54:48.240508 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:48.240337 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" podUID="f3b08299-0d08-470e-ab89-d77ecd29a504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 16:54:49.240504 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:49.240471 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 16:54:49.240941 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:49.240553 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" podUID="f3b08299-0d08-470e-ab89-d77ecd29a504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 16:54:50.245340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:50.245302 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:54:50.245875 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:50.245779 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:54:50.246547 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:54:50.246514 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 16:54:50.291702 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:54:50.291661 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:55:20.291144 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:55:20.291105 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:55:30.757942 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:30.757906 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf"] Oct 30 16:55:30.763525 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:30.763507 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" Oct 30 16:55:30.783129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:30.783108 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf"] Oct 30 16:55:30.791204 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:30.791173 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-06e7f6\"" Oct 30 16:55:30.791457 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:30.791433 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-06e7f6-dockercfg-hrhzp\"" Oct 30 16:55:30.937743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:30.930110 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/932e0df6-6b3d-4904-a353-ae7faf5d2544-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf\" (UID: \"932e0df6-6b3d-4904-a353-ae7faf5d2544\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" Oct 30 16:55:30.938131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:30.938102 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp2jl\" (UniqueName: \"kubernetes.io/projected/932e0df6-6b3d-4904-a353-ae7faf5d2544-kube-api-access-mp2jl\") pod \"payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf\" (UID: \"932e0df6-6b3d-4904-a353-ae7faf5d2544\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" Oct 30 16:55:31.042151 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.042057 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/932e0df6-6b3d-4904-a353-ae7faf5d2544-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf\" (UID: \"932e0df6-6b3d-4904-a353-ae7faf5d2544\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" Oct 30 16:55:31.043276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.043239 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp2jl\" (UniqueName: \"kubernetes.io/projected/932e0df6-6b3d-4904-a353-ae7faf5d2544-kube-api-access-mp2jl\") pod \"payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf\" (UID: \"932e0df6-6b3d-4904-a353-ae7faf5d2544\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" Oct 30 16:55:31.045023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.044969 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/932e0df6-6b3d-4904-a353-ae7faf5d2544-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf\" (UID: \"932e0df6-6b3d-4904-a353-ae7faf5d2544\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" Oct 30 16:55:31.049111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.049082 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d"] Oct 30 16:55:31.070688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.069696 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d"] Oct 30 16:55:31.070688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.069957 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" Oct 30 16:55:31.070961 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.070864 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp2jl\" (UniqueName: \"kubernetes.io/projected/932e0df6-6b3d-4904-a353-ae7faf5d2544-kube-api-access-mp2jl\") pod \"payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf\" (UID: \"932e0df6-6b3d-4904-a353-ae7faf5d2544\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" Oct 30 16:55:31.090473 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.090435 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" Oct 30 16:55:31.145250 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.145218 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz8g4\" (UniqueName: \"kubernetes.io/projected/71e9a70b-d593-49ba-82d4-0a022e09197a-kube-api-access-bz8g4\") pod \"payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d\" (UID: \"71e9a70b-d593-49ba-82d4-0a022e09197a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" Oct 30 16:55:31.145438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.145384 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e9a70b-d593-49ba-82d4-0a022e09197a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d\" (UID: \"71e9a70b-d593-49ba-82d4-0a022e09197a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" Oct 30 16:55:31.246944 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.246912 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e9a70b-d593-49ba-82d4-0a022e09197a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d\" (UID: \"71e9a70b-d593-49ba-82d4-0a022e09197a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" Oct 30 16:55:31.247159 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.247141 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz8g4\" (UniqueName: \"kubernetes.io/projected/71e9a70b-d593-49ba-82d4-0a022e09197a-kube-api-access-bz8g4\") pod \"payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d\" (UID: \"71e9a70b-d593-49ba-82d4-0a022e09197a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" Oct 30 16:55:31.247903 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.247881 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e9a70b-d593-49ba-82d4-0a022e09197a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d\" (UID: \"71e9a70b-d593-49ba-82d4-0a022e09197a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" Oct 30 16:55:31.271442 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.271411 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz8g4\" (UniqueName: \"kubernetes.io/projected/71e9a70b-d593-49ba-82d4-0a022e09197a-kube-api-access-bz8g4\") pod \"payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d\" (UID: \"71e9a70b-d593-49ba-82d4-0a022e09197a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" Oct 30 16:55:31.412107 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.412076 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf"] Oct 30 16:55:31.412590 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:55:31.412560 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932e0df6_6b3d_4904_a353_ae7faf5d2544.slice/crio-8c3c3844b0d4123357d10e51c11946a690d723d3fc7d3bf9c4f3e4dc88b15564 WatchSource:0}: Error finding container 8c3c3844b0d4123357d10e51c11946a690d723d3fc7d3bf9c4f3e4dc88b15564: Status 404 returned error can't find the container with id 8c3c3844b0d4123357d10e51c11946a690d723d3fc7d3bf9c4f3e4dc88b15564 Oct 30 16:55:31.414229 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.414158 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" Oct 30 16:55:31.544796 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.544755 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" event={"ID":"932e0df6-6b3d-4904-a353-ae7faf5d2544","Type":"ContainerStarted","Data":"6fde81dc89bdfb504e5f0747ae3224d8595625219f70f5ebfe6f68ee23b7dc7f"} Oct 30 16:55:31.545001 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.544826 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" event={"ID":"932e0df6-6b3d-4904-a353-ae7faf5d2544","Type":"ContainerStarted","Data":"8c3c3844b0d4123357d10e51c11946a690d723d3fc7d3bf9c4f3e4dc88b15564"} Oct 30 16:55:31.545856 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.545830 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" Oct 30 16:55:31.589666 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.589619 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" podStartSLOduration=1.589604548 podStartE2EDuration="1.589604548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:31.587442767 +0000 UTC m=+1961.930116805" watchObservedRunningTime="2026-10-30 16:55:31.589604548 +0000 UTC m=+1961.932278606" Oct 30 16:55:31.768940 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:31.768133 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d"] Oct 30 16:55:31.780540 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:55:31.780508 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e9a70b_d593_49ba_82d4_0a022e09197a.slice/crio-bc5401c10a6b0d6af4226024d876bc8f76c97ae2c098f2c20cd59c73c0838127 WatchSource:0}: Error finding container bc5401c10a6b0d6af4226024d876bc8f76c97ae2c098f2c20cd59c73c0838127: Status 404 returned error can't find the container with id bc5401c10a6b0d6af4226024d876bc8f76c97ae2c098f2c20cd59c73c0838127 Oct 30 16:55:32.553755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:32.553684 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" event={"ID":"71e9a70b-d593-49ba-82d4-0a022e09197a","Type":"ContainerStarted","Data":"3773b154f04e6864172825bc04faad2051a7d8baae036b6e078f25c8340bccec"} Oct 30 16:55:32.553965 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:32.553939 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" Oct 30 16:55:32.554023 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:32.553978 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" event={"ID":"71e9a70b-d593-49ba-82d4-0a022e09197a","Type":"ContainerStarted","Data":"bc5401c10a6b0d6af4226024d876bc8f76c97ae2c098f2c20cd59c73c0838127"} Oct 30 16:55:32.595878 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:32.595832 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" podStartSLOduration=2.595820387 podStartE2EDuration="2.595820387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:32.595405882 +0000 UTC m=+1962.938079920" watchObservedRunningTime="2026-10-30 16:55:32.595820387 +0000 UTC m=+1962.938494424" Oct 30 16:55:38.920392 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:38.920357 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz"] Oct 30 16:55:43.564416 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:43.564387 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" Oct 30 16:55:43.564901 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:43.564812 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" Oct 30 16:55:43.920805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:43.920714 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn"] Oct 30 16:55:43.920805 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:43.920750 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" podUID="033a9d36-1eef-48f4-8615-f96fa569eee5" containerName="payload-pre-processing" containerID="cri-o://86e21b3c15d997e93a7d54ace82e56f66f9c4a80ecea7bd57c620fca78f6a3df" gracePeriod=25 Oct 30 16:55:43.925562 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:43.925540 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:43.944788 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:43.944693 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-cf4621\"/\"openshift-service-ca.crt\"" Oct 30 16:55:43.945183 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:43.945148 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-cf4621\"/\"default-dockercfg-59q79\"" Oct 30 16:55:43.945863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:43.944958 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-cf4621\"/\"auth-test-model-cf4621-kserve-self-signed-certs\"" Oct 30 16:55:43.952479 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:43.952449 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn"] Oct 30 16:55:43.954009 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:43.953987 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-cf4621\"/\"kube-root-ca.crt\"" Oct 30 16:55:44.084260 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.084225 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-home\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.084414 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.084286 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-tmp-dir\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.084414 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.084348 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8rnh\" (UniqueName: \"kubernetes.io/projected/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-kube-api-access-j8rnh\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.084414 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.084378 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-model-cache\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.084547 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.084429 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-dshm\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.084547 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.084492 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-tls-certs\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.199475 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.199374 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-home\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.199475 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.199462 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-tmp-dir\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.199740 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.199489 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8rnh\" (UniqueName: \"kubernetes.io/projected/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-kube-api-access-j8rnh\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.199740 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.199528 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-model-cache\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.199740 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.199595 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-dshm\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.199740 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.199620 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-tls-certs\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.213090 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.200474 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-home\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.213507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.213477 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-tmp-dir\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.216919 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.216891 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-model-cache\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.244515 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.244410 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-tls-certs\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.244764 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.244715 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-dshm\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.287893 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.287858 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8rnh\" (UniqueName: \"kubernetes.io/projected/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-kube-api-access-j8rnh\") pod \"auth-test-model-cf4621-kserve-85b4f6dddf-n27fn\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.556305 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.556272 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:44.796894 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.796850 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" podUID="033a9d36-1eef-48f4-8615-f96fa569eee5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:55:44.870472 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:44.870445 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn"] Oct 30 16:55:44.870959 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:55:44.870933 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95967951_8ac6_4287_8b0f_9f4a91a6ac0f.slice/crio-4d71a13190905657b1c4930340540ff595c532fe875ffe744a0b49829b1a4480 WatchSource:0}: Error finding container 4d71a13190905657b1c4930340540ff595c532fe875ffe744a0b49829b1a4480: Status 404 returned error can't find the container with id 4d71a13190905657b1c4930340540ff595c532fe875ffe744a0b49829b1a4480 Oct 30 16:55:45.678345 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:45.678304 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" event={"ID":"95967951-8ac6-4287-8b0f-9f4a91a6ac0f","Type":"ContainerStarted","Data":"efbbb811180ae2b533106299d8b5a5cf1ca3f03afcf8b167a2a2f6d1ee02974e"} Oct 30 16:55:45.678518 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:45.678359 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" event={"ID":"95967951-8ac6-4287-8b0f-9f4a91a6ac0f","Type":"ContainerStarted","Data":"4d71a13190905657b1c4930340540ff595c532fe875ffe744a0b49829b1a4480"} Oct 30 16:55:45.724444 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:45.724379 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" podStartSLOduration=2.72436153 podStartE2EDuration="2.72436153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:45.7216951 +0000 UTC m=+1976.064369136" watchObservedRunningTime="2026-10-30 16:55:45.72436153 +0000 UTC m=+1976.067035569" Oct 30 16:55:46.557379 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:46.557352 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:50.303805 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:55:50.303753 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:55:54.556533 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:54.556499 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:54.570155 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:54.570117 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:54.756438 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:54.756409 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 16:55:54.780934 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:54.780907 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" podUID="033a9d36-1eef-48f4-8615-f96fa569eee5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:55:55.465588 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.465545 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf"] Oct 30 16:55:55.470442 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.470417 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 16:55:55.504677 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.500524 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-06e7f6-serving-cert\"" Oct 30 16:55:55.507305 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.507272 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf"] Oct 30 16:55:55.628966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.628937 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2087171b-22dc-444f-9f62-dd3003edbff9-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf\" (UID: \"2087171b-22dc-444f-9f62-dd3003edbff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 16:55:55.629363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.628972 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2087171b-22dc-444f-9f62-dd3003edbff9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf\" (UID: \"2087171b-22dc-444f-9f62-dd3003edbff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 16:55:55.629363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.629073 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7kvt\" (UniqueName: \"kubernetes.io/projected/2087171b-22dc-444f-9f62-dd3003edbff9-kube-api-access-j7kvt\") pod \"maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf\" (UID: \"2087171b-22dc-444f-9f62-dd3003edbff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 16:55:55.629363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.629199 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2087171b-22dc-444f-9f62-dd3003edbff9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf\" (UID: \"2087171b-22dc-444f-9f62-dd3003edbff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 16:55:55.736752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.732973 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2087171b-22dc-444f-9f62-dd3003edbff9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf\" (UID: \"2087171b-22dc-444f-9f62-dd3003edbff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 16:55:55.736752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.733141 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2087171b-22dc-444f-9f62-dd3003edbff9-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf\" (UID: \"2087171b-22dc-444f-9f62-dd3003edbff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 16:55:55.736752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.733176 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2087171b-22dc-444f-9f62-dd3003edbff9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf\" (UID: \"2087171b-22dc-444f-9f62-dd3003edbff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 16:55:55.736752 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.733258 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7kvt\" (UniqueName: \"kubernetes.io/projected/2087171b-22dc-444f-9f62-dd3003edbff9-kube-api-access-j7kvt\") pod \"maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf\" (UID: \"2087171b-22dc-444f-9f62-dd3003edbff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 16:55:55.738428 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.738379 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2087171b-22dc-444f-9f62-dd3003edbff9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf\" (UID: \"2087171b-22dc-444f-9f62-dd3003edbff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 16:55:55.781235 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.775159 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2087171b-22dc-444f-9f62-dd3003edbff9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf\" (UID: \"2087171b-22dc-444f-9f62-dd3003edbff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 16:55:55.781235 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.778713 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7kvt\" (UniqueName: \"kubernetes.io/projected/2087171b-22dc-444f-9f62-dd3003edbff9-kube-api-access-j7kvt\") pod \"maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf\" (UID: \"2087171b-22dc-444f-9f62-dd3003edbff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 16:55:55.781495 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.781435 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2087171b-22dc-444f-9f62-dd3003edbff9-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf\" (UID: \"2087171b-22dc-444f-9f62-dd3003edbff9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 16:55:55.785129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:55.785013 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 16:55:56.100362 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:56.100330 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf"] Oct 30 16:55:56.101275 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:55:56.101252 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2087171b_22dc_444f_9f62_dd3003edbff9.slice/crio-7552f3fc910846d3e1b55f33d42b71376f3abff72bff84e9797ff7bc2c353d46 WatchSource:0}: Error finding container 7552f3fc910846d3e1b55f33d42b71376f3abff72bff84e9797ff7bc2c353d46: Status 404 returned error can't find the container with id 7552f3fc910846d3e1b55f33d42b71376f3abff72bff84e9797ff7bc2c353d46 Oct 30 16:55:56.841801 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:56.840962 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" event={"ID":"2087171b-22dc-444f-9f62-dd3003edbff9","Type":"ContainerStarted","Data":"eb21274d861f8e00cde5d61b85b14db89a69d68848fe7392356af7431caf62f7"} Oct 30 16:55:56.841801 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:56.841035 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" event={"ID":"2087171b-22dc-444f-9f62-dd3003edbff9","Type":"ContainerStarted","Data":"7552f3fc910846d3e1b55f33d42b71376f3abff72bff84e9797ff7bc2c353d46"} Oct 30 16:55:56.842366 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:56.842296 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 16:55:56.922793 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:55:56.921979 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" podStartSLOduration=1.511889081 podStartE2EDuration="1.921955315s" totalImagesPullingTime="410.066234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:56.919690456 +0000 UTC m=+1987.262364495" watchObservedRunningTime="2026-10-30 16:55:56.921955315 +0000 UTC m=+1987.264629352" Oct 30 16:56:03.862149 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:03.862115 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 16:56:04.210680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:04.210651 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" Oct 30 16:56:04.301816 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:04.301780 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033a9d36-1eef-48f4-8615-f96fa569eee5-kube-api-access-trmzx\" (UniqueName: \"kubernetes.io/projected/033a9d36-1eef-48f4-8615-f96fa569eee5-kube-api-access-trmzx\") pod \"033a9d36-1eef-48f4-8615-f96fa569eee5\" (UID: \"033a9d36-1eef-48f4-8615-f96fa569eee5\") " Oct 30 16:56:04.302008 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:04.301876 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033a9d36-1eef-48f4-8615-f96fa569eee5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033a9d36-1eef-48f4-8615-f96fa569eee5-plugins-config-volume\") pod \"033a9d36-1eef-48f4-8615-f96fa569eee5\" (UID: \"033a9d36-1eef-48f4-8615-f96fa569eee5\") " Oct 30 16:56:04.302236 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:04.302212 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033a9d36-1eef-48f4-8615-f96fa569eee5-plugins-config-volume" pod "033a9d36-1eef-48f4-8615-f96fa569eee5" (UID: "033a9d36-1eef-48f4-8615-f96fa569eee5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:56:04.302480 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:04.302460 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/033a9d36-1eef-48f4-8615-f96fa569eee5-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:56:04.304313 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:04.304283 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033a9d36-1eef-48f4-8615-f96fa569eee5-kube-api-access-trmzx" pod "033a9d36-1eef-48f4-8615-f96fa569eee5" (UID: "033a9d36-1eef-48f4-8615-f96fa569eee5"). InnerVolumeSpecName "kube-api-access-trmzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:04.404475 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:04.404444 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trmzx\" (UniqueName: \"kubernetes.io/projected/033a9d36-1eef-48f4-8615-f96fa569eee5-kube-api-access-trmzx\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:56:04.896802 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:04.896765 4861 generic.go:356] "Generic (PLEG): container finished" podID="033a9d36-1eef-48f4-8615-f96fa569eee5" containerID="86e21b3c15d997e93a7d54ace82e56f66f9c4a80ecea7bd57c620fca78f6a3df" exitCode=0 Oct 30 16:56:04.897266 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:04.896849 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" Oct 30 16:56:04.897266 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:04.896881 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" event={"ID":"033a9d36-1eef-48f4-8615-f96fa569eee5","Type":"ContainerDied","Data":"86e21b3c15d997e93a7d54ace82e56f66f9c4a80ecea7bd57c620fca78f6a3df"} Oct 30 16:56:04.897266 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:04.896938 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz" event={"ID":"033a9d36-1eef-48f4-8615-f96fa569eee5","Type":"ContainerDied","Data":"feb77340abe005af250cd298abdfdda6494110da6b7fcc5742827a3b19f27f3c"} Oct 30 16:56:04.897266 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:04.896960 4861 scope.go:122] "RemoveContainer" containerID="86e21b3c15d997e93a7d54ace82e56f66f9c4a80ecea7bd57c620fca78f6a3df" Oct 30 16:56:04.908340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:04.908322 4861 scope.go:122] "RemoveContainer" containerID="86e21b3c15d997e93a7d54ace82e56f66f9c4a80ecea7bd57c620fca78f6a3df" Oct 30 16:56:04.908614 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:56:04.908596 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e21b3c15d997e93a7d54ace82e56f66f9c4a80ecea7bd57c620fca78f6a3df\": container with ID starting with 86e21b3c15d997e93a7d54ace82e56f66f9c4a80ecea7bd57c620fca78f6a3df not found: ID does not exist" containerID="86e21b3c15d997e93a7d54ace82e56f66f9c4a80ecea7bd57c620fca78f6a3df" Oct 30 16:56:04.908662 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:04.908621 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e21b3c15d997e93a7d54ace82e56f66f9c4a80ecea7bd57c620fca78f6a3df"} err="failed to get container status \"86e21b3c15d997e93a7d54ace82e56f66f9c4a80ecea7bd57c620fca78f6a3df\": rpc error: code = NotFound desc = could not find container \"86e21b3c15d997e93a7d54ace82e56f66f9c4a80ecea7bd57c620fca78f6a3df\": container with ID starting with 86e21b3c15d997e93a7d54ace82e56f66f9c4a80ecea7bd57c620fca78f6a3df not found: ID does not exist" Oct 30 16:56:04.990887 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:04.990854 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz"] Oct 30 16:56:04.994533 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:04.994504 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-07cb3c-5b77f88975-wllfz"] Oct 30 16:56:05.808393 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:05.808354 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033a9d36-1eef-48f4-8615-f96fa569eee5" path="/var/lib/kubelet/pods/033a9d36-1eef-48f4-8615-f96fa569eee5/volumes" Oct 30 16:56:20.290822 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:56:20.290782 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:56:37.768580 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:37.767577 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k"] Oct 30 16:56:37.768580 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:37.768160 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" podUID="1a9f9f36-9d38-42ae-a42b-014d740d3729" containerName="maas-api" containerID="cri-o://91992ca98c47ab3ec61d10c99e7e3befdad4dcb020247318ed1df2ee0a740e7b" gracePeriod=30 Oct 30 16:56:37.864481 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:56:37.864447 4861 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a9f9f36_9d38_42ae_a42b_014d740d3729.slice/crio-conmon-91992ca98c47ab3ec61d10c99e7e3befdad4dcb020247318ed1df2ee0a740e7b.scope\": RecentStats: unable to find data in memory cache]" Oct 30 16:56:38.074299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.074275 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:56:38.204712 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.204675 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a9f9f36-9d38-42ae-a42b-014d740d3729-kube-api-access-jlg9l\" (UniqueName: \"kubernetes.io/projected/1a9f9f36-9d38-42ae-a42b-014d740d3729-kube-api-access-jlg9l\") pod \"1a9f9f36-9d38-42ae-a42b-014d740d3729\" (UID: \"1a9f9f36-9d38-42ae-a42b-014d740d3729\") " Oct 30 16:56:38.204897 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.204831 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a9f9f36-9d38-42ae-a42b-014d740d3729-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9f9f36-9d38-42ae-a42b-014d740d3729-maas-api-tls\") pod \"1a9f9f36-9d38-42ae-a42b-014d740d3729\" (UID: \"1a9f9f36-9d38-42ae-a42b-014d740d3729\") " Oct 30 16:56:38.204967 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.204950 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a9f9f36-9d38-42ae-a42b-014d740d3729-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9f9f36-9d38-42ae-a42b-014d740d3729-metrics-certs\") pod \"1a9f9f36-9d38-42ae-a42b-014d740d3729\" (UID: \"1a9f9f36-9d38-42ae-a42b-014d740d3729\") " Oct 30 16:56:38.205021 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.204989 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a9f9f36-9d38-42ae-a42b-014d740d3729-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f9f36-9d38-42ae-a42b-014d740d3729-openshift-service-ca\") pod \"1a9f9f36-9d38-42ae-a42b-014d740d3729\" (UID: \"1a9f9f36-9d38-42ae-a42b-014d740d3729\") " Oct 30 16:56:38.205622 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.205588 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a9f9f36-9d38-42ae-a42b-014d740d3729-openshift-service-ca" pod "1a9f9f36-9d38-42ae-a42b-014d740d3729" (UID: "1a9f9f36-9d38-42ae-a42b-014d740d3729"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:56:38.207343 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.207313 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a9f9f36-9d38-42ae-a42b-014d740d3729-kube-api-access-jlg9l" pod "1a9f9f36-9d38-42ae-a42b-014d740d3729" (UID: "1a9f9f36-9d38-42ae-a42b-014d740d3729"). InnerVolumeSpecName "kube-api-access-jlg9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:38.207343 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.207323 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9f9f36-9d38-42ae-a42b-014d740d3729-metrics-certs" pod "1a9f9f36-9d38-42ae-a42b-014d740d3729" (UID: "1a9f9f36-9d38-42ae-a42b-014d740d3729"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:56:38.207696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.207673 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a9f9f36-9d38-42ae-a42b-014d740d3729-maas-api-tls" pod "1a9f9f36-9d38-42ae-a42b-014d740d3729" (UID: "1a9f9f36-9d38-42ae-a42b-014d740d3729"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:56:38.215419 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.215390 4861 generic.go:356] "Generic (PLEG): container finished" podID="1a9f9f36-9d38-42ae-a42b-014d740d3729" containerID="91992ca98c47ab3ec61d10c99e7e3befdad4dcb020247318ed1df2ee0a740e7b" exitCode=0 Oct 30 16:56:38.215549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.215444 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" Oct 30 16:56:38.215549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.215474 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" event={"ID":"1a9f9f36-9d38-42ae-a42b-014d740d3729","Type":"ContainerDied","Data":"91992ca98c47ab3ec61d10c99e7e3befdad4dcb020247318ed1df2ee0a740e7b"} Oct 30 16:56:38.215549 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.215538 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k" event={"ID":"1a9f9f36-9d38-42ae-a42b-014d740d3729","Type":"ContainerDied","Data":"71d70c34fdf35d3b5cadf59003c725c2ef5d843492fae1ff103063b64c94f4a3"} Oct 30 16:56:38.215747 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.215571 4861 scope.go:122] "RemoveContainer" containerID="91992ca98c47ab3ec61d10c99e7e3befdad4dcb020247318ed1df2ee0a740e7b" Oct 30 16:56:38.250202 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.250182 4861 scope.go:122] "RemoveContainer" containerID="91992ca98c47ab3ec61d10c99e7e3befdad4dcb020247318ed1df2ee0a740e7b" Oct 30 16:56:38.250809 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:56:38.250745 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91992ca98c47ab3ec61d10c99e7e3befdad4dcb020247318ed1df2ee0a740e7b\": container with ID starting with 91992ca98c47ab3ec61d10c99e7e3befdad4dcb020247318ed1df2ee0a740e7b not found: ID does not exist" containerID="91992ca98c47ab3ec61d10c99e7e3befdad4dcb020247318ed1df2ee0a740e7b" Oct 30 16:56:38.250809 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.250781 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91992ca98c47ab3ec61d10c99e7e3befdad4dcb020247318ed1df2ee0a740e7b"} err="failed to get container status \"91992ca98c47ab3ec61d10c99e7e3befdad4dcb020247318ed1df2ee0a740e7b\": rpc error: code = NotFound desc = could not find container \"91992ca98c47ab3ec61d10c99e7e3befdad4dcb020247318ed1df2ee0a740e7b\": container with ID starting with 91992ca98c47ab3ec61d10c99e7e3befdad4dcb020247318ed1df2ee0a740e7b not found: ID does not exist" Oct 30 16:56:38.314904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.307318 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a9f9f36-9d38-42ae-a42b-014d740d3729-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:56:38.314904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.307354 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a9f9f36-9d38-42ae-a42b-014d740d3729-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:56:38.314904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.307370 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlg9l\" (UniqueName: \"kubernetes.io/projected/1a9f9f36-9d38-42ae-a42b-014d740d3729-kube-api-access-jlg9l\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:56:38.314904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.307384 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a9f9f36-9d38-42ae-a42b-014d740d3729-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:56:38.336453 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.336409 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k"] Oct 30 16:56:38.370115 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:38.370078 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-07cb3c-84c955d84b-9697k"] Oct 30 16:56:39.817601 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:39.817566 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a9f9f36-9d38-42ae-a42b-014d740d3729" path="/var/lib/kubelet/pods/1a9f9f36-9d38-42ae-a42b-014d740d3729/volumes" Oct 30 16:56:44.039831 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:44.039801 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp"] Oct 30 16:56:50.289583 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:56:50.289545 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:56:50.848514 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.848480 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq"] Oct 30 16:56:50.849232 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.849213 4861 cpu_manager.go:414] "containerMap: removing container" podUID="033a9d36-1eef-48f4-8615-f96fa569eee5" containerName="payload-pre-processing" Oct 30 16:56:50.849295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.849233 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033a9d36-1eef-48f4-8615-f96fa569eee5" containerName="payload-pre-processing" Oct 30 16:56:50.849295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.849264 4861 cpu_manager.go:414] "containerMap: removing container" podUID="1a9f9f36-9d38-42ae-a42b-014d740d3729" containerName="maas-api" Oct 30 16:56:50.849295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.849269 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a9f9f36-9d38-42ae-a42b-014d740d3729" containerName="maas-api" Oct 30 16:56:50.849437 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.849415 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="033a9d36-1eef-48f4-8615-f96fa569eee5" containerName="payload-pre-processing" Oct 30 16:56:50.849437 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.849428 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a9f9f36-9d38-42ae-a42b-014d740d3729" containerName="maas-api" Oct 30 16:56:50.853523 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.853499 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:50.876294 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.876268 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq"] Oct 30 16:56:50.883300 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.883271 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-799c76\"/\"kube-root-ca.crt\"" Oct 30 16:56:50.883447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.883379 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-799c76\"/\"openshift-service-ca.crt\"" Oct 30 16:56:50.883527 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.883509 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-799c76\"/\"auth-test-model-799c76-kserve-self-signed-certs\"" Oct 30 16:56:50.883587 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.883575 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-799c76\"/\"default-dockercfg-97msb\"" Oct 30 16:56:50.917273 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.917246 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-tls-certs\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:50.917434 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.917310 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htwxd\" (UniqueName: \"kubernetes.io/projected/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-kube-api-access-htwxd\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:50.917582 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.917560 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-model-cache\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:50.917650 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.917633 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-home\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:50.917846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.917822 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-tmp-dir\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:50.918005 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:50.917987 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-dshm\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:51.019580 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:51.019548 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-model-cache\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:51.019776 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:51.019600 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-home\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:51.019776 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:51.019641 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-tmp-dir\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:51.019776 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:51.019710 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-dshm\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:51.019993 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:51.019783 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-tls-certs\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:51.019993 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:51.019831 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htwxd\" (UniqueName: \"kubernetes.io/projected/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-kube-api-access-htwxd\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:51.020129 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:51.020056 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-model-cache\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:51.020265 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:51.020222 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-home\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:51.020371 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:51.020248 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-tmp-dir\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:51.022436 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:51.022411 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-dshm\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:51.022715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:51.022690 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-tls-certs\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:51.044272 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:51.044245 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htwxd\" (UniqueName: \"kubernetes.io/projected/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-kube-api-access-htwxd\") pod \"auth-test-model-799c76-kserve-5c48df4649-9d8gq\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:51.184800 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:51.169248 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:51.578060 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:51.578027 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq"] Oct 30 16:56:51.578414 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:56:51.578282 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc1a58ac_4a05_4766_b212_7eff1c8c0e7f.slice/crio-03fb80be8ebe87bbfc70c1cc420f5568bad319bc547c99233f5eda1cbe654094 WatchSource:0}: Error finding container 03fb80be8ebe87bbfc70c1cc420f5568bad319bc547c99233f5eda1cbe654094: Status 404 returned error can't find the container with id 03fb80be8ebe87bbfc70c1cc420f5568bad319bc547c99233f5eda1cbe654094 Oct 30 16:56:52.343337 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:52.343295 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" event={"ID":"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f","Type":"ContainerStarted","Data":"db9b896421464da50665563bfc4161046ba77b188c0a974f7137cf091c42b37d"} Oct 30 16:56:52.343533 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:52.343349 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" event={"ID":"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f","Type":"ContainerStarted","Data":"03fb80be8ebe87bbfc70c1cc420f5568bad319bc547c99233f5eda1cbe654094"} Oct 30 16:56:52.388854 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:52.388798 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" podStartSLOduration=2.388772377 podStartE2EDuration="2.388772377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:56:52.384875125 +0000 UTC m=+2042.727549188" watchObservedRunningTime="2026-10-30 16:56:52.388772377 +0000 UTC m=+2042.731446415" Oct 30 16:56:53.170405 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:53.170376 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:56:59.069753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.066205 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" podUID="b0b17bec-f0b9-492a-b059-fa50daf2ef63" containerName="main" containerID="cri-o://bbfccc005b2a422998df26f74ac8d08fad444e74921371b23750b6409856d3e0" gracePeriod=45 Oct 30 16:56:59.384846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.384822 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:56:59.432706 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.432670 4861 generic.go:356] "Generic (PLEG): container finished" podID="b0b17bec-f0b9-492a-b059-fa50daf2ef63" containerID="bbfccc005b2a422998df26f74ac8d08fad444e74921371b23750b6409856d3e0" exitCode=0 Oct 30 16:56:59.432880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.432710 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" Oct 30 16:56:59.432880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.432754 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" event={"ID":"b0b17bec-f0b9-492a-b059-fa50daf2ef63","Type":"ContainerDied","Data":"bbfccc005b2a422998df26f74ac8d08fad444e74921371b23750b6409856d3e0"} Oct 30 16:56:59.432880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.432796 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp" event={"ID":"b0b17bec-f0b9-492a-b059-fa50daf2ef63","Type":"ContainerDied","Data":"efc91bfe7e6027a68f9bc62028e9d43e0f44d6f8af006befc089151ce690923f"} Oct 30 16:56:59.432880 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.432818 4861 scope.go:122] "RemoveContainer" containerID="bbfccc005b2a422998df26f74ac8d08fad444e74921371b23750b6409856d3e0" Oct 30 16:56:59.446501 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.446473 4861 scope.go:122] "RemoveContainer" containerID="bbfccc005b2a422998df26f74ac8d08fad444e74921371b23750b6409856d3e0" Oct 30 16:56:59.446910 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:56:59.446885 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbfccc005b2a422998df26f74ac8d08fad444e74921371b23750b6409856d3e0\": container with ID starting with bbfccc005b2a422998df26f74ac8d08fad444e74921371b23750b6409856d3e0 not found: ID does not exist" containerID="bbfccc005b2a422998df26f74ac8d08fad444e74921371b23750b6409856d3e0" Oct 30 16:56:59.447055 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.446922 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbfccc005b2a422998df26f74ac8d08fad444e74921371b23750b6409856d3e0"} err="failed to get container status \"bbfccc005b2a422998df26f74ac8d08fad444e74921371b23750b6409856d3e0\": rpc error: code = NotFound desc = could not find container \"bbfccc005b2a422998df26f74ac8d08fad444e74921371b23750b6409856d3e0\": container with ID starting with bbfccc005b2a422998df26f74ac8d08fad444e74921371b23750b6409856d3e0 not found: ID does not exist" Oct 30 16:56:59.489951 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.489924 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-home\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-home\") pod \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " Oct 30 16:56:59.490118 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.489974 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0b17bec-f0b9-492a-b059-fa50daf2ef63-tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b17bec-f0b9-492a-b059-fa50daf2ef63-tls-certs\") pod \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " Oct 30 16:56:59.490223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.490121 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b17bec-f0b9-492a-b059-fa50daf2ef63-kube-api-access-w7hcl\" (UniqueName: \"kubernetes.io/projected/b0b17bec-f0b9-492a-b059-fa50daf2ef63-kube-api-access-w7hcl\") pod \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " Oct 30 16:56:59.490223 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.490168 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-model-cache\") pod \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " Oct 30 16:56:59.490339 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.490212 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-home" pod "b0b17bec-f0b9-492a-b059-fa50daf2ef63" (UID: "b0b17bec-f0b9-492a-b059-fa50daf2ef63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:59.490339 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.490281 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-tmp-dir\") pod \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " Oct 30 16:56:59.490481 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.490464 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-dshm\") pod \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\" (UID: \"b0b17bec-f0b9-492a-b059-fa50daf2ef63\") " Oct 30 16:56:59.490591 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.490464 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-model-cache" pod "b0b17bec-f0b9-492a-b059-fa50daf2ef63" (UID: "b0b17bec-f0b9-492a-b059-fa50daf2ef63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:59.490649 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.490556 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-tmp-dir" pod "b0b17bec-f0b9-492a-b059-fa50daf2ef63" (UID: "b0b17bec-f0b9-492a-b059-fa50daf2ef63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:59.491383 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.491356 4861 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:56:59.491497 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.491386 4861 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:56:59.491497 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.491400 4861 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:56:59.492629 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.492600 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-dshm" pod "b0b17bec-f0b9-492a-b059-fa50daf2ef63" (UID: "b0b17bec-f0b9-492a-b059-fa50daf2ef63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:59.492629 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.492606 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b17bec-f0b9-492a-b059-fa50daf2ef63-kube-api-access-w7hcl" pod "b0b17bec-f0b9-492a-b059-fa50daf2ef63" (UID: "b0b17bec-f0b9-492a-b059-fa50daf2ef63"). InnerVolumeSpecName "kube-api-access-w7hcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:59.492814 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.492700 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b17bec-f0b9-492a-b059-fa50daf2ef63-tls-certs" pod "b0b17bec-f0b9-492a-b059-fa50daf2ef63" (UID: "b0b17bec-f0b9-492a-b059-fa50daf2ef63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:56:59.592958 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.592926 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7hcl\" (UniqueName: \"kubernetes.io/projected/b0b17bec-f0b9-492a-b059-fa50daf2ef63-kube-api-access-w7hcl\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:56:59.592958 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.592951 4861 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0b17bec-f0b9-492a-b059-fa50daf2ef63-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:56:59.592958 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.592961 4861 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0b17bec-f0b9-492a-b059-fa50daf2ef63-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:56:59.780634 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.780605 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp"] Oct 30 16:56:59.787125 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.787102 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-07cb3c/praxis-route-07cb3c-kserve-545b66c4d5-4r6gp"] Oct 30 16:56:59.808507 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:56:59.808471 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b17bec-f0b9-492a-b059-fa50daf2ef63" path="/var/lib/kubelet/pods/b0b17bec-f0b9-492a-b059-fa50daf2ef63/volumes" Oct 30 16:57:01.169768 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:01.169708 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:57:01.182808 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:01.182786 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:57:01.459695 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:01.459609 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 16:57:09.265295 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.265259 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d"] Oct 30 16:57:09.266005 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.265991 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b0b17bec-f0b9-492a-b059-fa50daf2ef63" containerName="main" Oct 30 16:57:09.266005 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.266006 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b17bec-f0b9-492a-b059-fa50daf2ef63" containerName="main" Oct 30 16:57:09.266170 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.266130 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b17bec-f0b9-492a-b059-fa50daf2ef63" containerName="main" Oct 30 16:57:09.271181 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.271159 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.299149 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.299126 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d"] Oct 30 16:57:09.305824 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.305803 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-06e7f6\"/\"default-dockercfg-rrtzl\"" Oct 30 16:57:09.322738 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.322702 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-06e7f6\"/\"auth-test-model-06e7f6-kserve-self-signed-certs\"" Oct 30 16:57:09.324156 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.324114 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-06e7f6\"/\"kube-root-ca.crt\"" Oct 30 16:57:09.335131 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.335114 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-06e7f6\"/\"openshift-service-ca.crt\"" Oct 30 16:57:09.364264 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.364238 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-model-cache\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.364384 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.364281 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-dshm\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.364459 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.364419 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxs6b\" (UniqueName: \"kubernetes.io/projected/7bab337b-715f-4094-840e-75f1fba95a21-kube-api-access-mxs6b\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.364573 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.364555 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-home\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.364669 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.364655 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-tmp-dir\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.364774 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.364762 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bab337b-715f-4094-840e-75f1fba95a21-tls-certs\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.466501 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.466467 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-home\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.466666 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.466609 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-tmp-dir\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.466783 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.466759 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bab337b-715f-4094-840e-75f1fba95a21-tls-certs\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.466859 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.466796 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-model-cache\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.466859 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.466825 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-dshm\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.466974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.466900 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxs6b\" (UniqueName: \"kubernetes.io/projected/7bab337b-715f-4094-840e-75f1fba95a21-kube-api-access-mxs6b\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.466974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.466924 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-home\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.467094 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.466996 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-tmp-dir\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.467236 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.467216 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-model-cache\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.469309 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.469271 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-dshm\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.469605 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.469585 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bab337b-715f-4094-840e-75f1fba95a21-tls-certs\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.493543 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.493510 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxs6b\" (UniqueName: \"kubernetes.io/projected/7bab337b-715f-4094-840e-75f1fba95a21-kube-api-access-mxs6b\") pod \"auth-test-model-06e7f6-kserve-84c5b44965-h459d\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:09.593797 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:09.585993 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:10.107985 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:10.107951 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d"] Oct 30 16:57:10.108380 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:57:10.108358 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bab337b_715f_4094_840e_75f1fba95a21.slice/crio-d68591efeac77175a958e5eb34bda34f23908b616a8ae699f28adb49e3a64597 WatchSource:0}: Error finding container d68591efeac77175a958e5eb34bda34f23908b616a8ae699f28adb49e3a64597: Status 404 returned error can't find the container with id d68591efeac77175a958e5eb34bda34f23908b616a8ae699f28adb49e3a64597 Oct 30 16:57:10.587815 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:10.585848 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" event={"ID":"7bab337b-715f-4094-840e-75f1fba95a21","Type":"ContainerStarted","Data":"38f2f8052ceff40ddcf044ae18729dd3f76491b3438ee3e388f274198f0bfe85"} Oct 30 16:57:10.587815 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:10.585908 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" event={"ID":"7bab337b-715f-4094-840e-75f1fba95a21","Type":"ContainerStarted","Data":"d68591efeac77175a958e5eb34bda34f23908b616a8ae699f28adb49e3a64597"} Oct 30 16:57:10.587815 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:10.586478 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:19.586715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:19.586672 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:19.599574 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:19.599544 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:19.642868 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:19.642816 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" podStartSLOduration=10.642801787 podStartE2EDuration="10.642801787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:10.635178681 +0000 UTC m=+2060.977852719" watchObservedRunningTime="2026-10-30 16:57:19.642801787 +0000 UTC m=+2069.985475850" Oct 30 16:57:19.667275 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:19.667244 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 16:57:20.291355 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:57:20.291316 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:57:50.139819 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:50.139792 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:57:50.165446 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:50.165424 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 16:57:50.291218 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:57:50.291184 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:57:56.914621 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:56.914578 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m"] Oct 30 16:57:56.949966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:57:56.949936 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9"] Oct 30 16:58:01.915792 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:01.915678 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" podUID="81d0fc53-fb6b-4437-a486-bb8ab2e3e39a" containerName="payload-processing" containerID="cri-o://b317c83873850bb81d69281275c4399e9c7db1b64a84c07a59a0ce5959c30a88" gracePeriod=25 Oct 30 16:58:01.951133 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:01.951081 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" podUID="5d066599-937f-44b9-89d0-dadc7d7478e9" containerName="payload-pre-processing" containerID="cri-o://25e082e239af41b617a26f28a0ea7d4213bf497f083660fd530ea406ce4478b2" gracePeriod=25 Oct 30 16:58:02.059044 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:58:02.059003 4861 logging.go:55] [core] [Channel #4868 SubChannel #4869]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.180:50052", ServerName: "10.132.0.180:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.180:50052: connect: connection refused" Oct 30 16:58:02.062386 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:02.062360 4861 generic.go:356] "Generic (PLEG): container finished" podID="81d0fc53-fb6b-4437-a486-bb8ab2e3e39a" containerID="b317c83873850bb81d69281275c4399e9c7db1b64a84c07a59a0ce5959c30a88" exitCode=0 Oct 30 16:58:02.062524 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:02.062441 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" event={"ID":"81d0fc53-fb6b-4437-a486-bb8ab2e3e39a","Type":"ContainerDied","Data":"b317c83873850bb81d69281275c4399e9c7db1b64a84c07a59a0ce5959c30a88"} Oct 30 16:58:02.241327 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:02.241302 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" Oct 30 16:58:02.275518 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:02.275495 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" Oct 30 16:58:02.356350 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:02.356313 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d0fc53-fb6b-4437-a486-bb8ab2e3e39a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d0fc53-fb6b-4437-a486-bb8ab2e3e39a-plugins-config-volume\") pod \"81d0fc53-fb6b-4437-a486-bb8ab2e3e39a\" (UID: \"81d0fc53-fb6b-4437-a486-bb8ab2e3e39a\") " Oct 30 16:58:02.356531 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:02.356417 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d0fc53-fb6b-4437-a486-bb8ab2e3e39a-kube-api-access-qf67s\" (UniqueName: \"kubernetes.io/projected/81d0fc53-fb6b-4437-a486-bb8ab2e3e39a-kube-api-access-qf67s\") pod \"81d0fc53-fb6b-4437-a486-bb8ab2e3e39a\" (UID: \"81d0fc53-fb6b-4437-a486-bb8ab2e3e39a\") " Oct 30 16:58:02.356585 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:02.356562 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d066599-937f-44b9-89d0-dadc7d7478e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d066599-937f-44b9-89d0-dadc7d7478e9-plugins-config-volume\") pod \"5d066599-937f-44b9-89d0-dadc7d7478e9\" (UID: \"5d066599-937f-44b9-89d0-dadc7d7478e9\") " Oct 30 16:58:02.356773 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:02.356703 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d066599-937f-44b9-89d0-dadc7d7478e9-kube-api-access-ptbs2\" (UniqueName: \"kubernetes.io/projected/5d066599-937f-44b9-89d0-dadc7d7478e9-kube-api-access-ptbs2\") pod \"5d066599-937f-44b9-89d0-dadc7d7478e9\" (UID: \"5d066599-937f-44b9-89d0-dadc7d7478e9\") " Oct 30 16:58:02.356956 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:02.356923 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d0fc53-fb6b-4437-a486-bb8ab2e3e39a-plugins-config-volume" pod "81d0fc53-fb6b-4437-a486-bb8ab2e3e39a" (UID: "81d0fc53-fb6b-4437-a486-bb8ab2e3e39a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:58:02.357099 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:02.356980 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d066599-937f-44b9-89d0-dadc7d7478e9-plugins-config-volume" pod "5d066599-937f-44b9-89d0-dadc7d7478e9" (UID: "5d066599-937f-44b9-89d0-dadc7d7478e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:58:02.357627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:02.357569 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d0fc53-fb6b-4437-a486-bb8ab2e3e39a-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:02.357627 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:02.357595 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d066599-937f-44b9-89d0-dadc7d7478e9-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:02.359203 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:02.359183 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d0fc53-fb6b-4437-a486-bb8ab2e3e39a-kube-api-access-qf67s" pod "81d0fc53-fb6b-4437-a486-bb8ab2e3e39a" (UID: "81d0fc53-fb6b-4437-a486-bb8ab2e3e39a"). InnerVolumeSpecName "kube-api-access-qf67s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:02.359540 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:02.359517 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d066599-937f-44b9-89d0-dadc7d7478e9-kube-api-access-ptbs2" pod "5d066599-937f-44b9-89d0-dadc7d7478e9" (UID: "5d066599-937f-44b9-89d0-dadc7d7478e9"). InnerVolumeSpecName "kube-api-access-ptbs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:02.458858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:02.458780 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptbs2\" (UniqueName: \"kubernetes.io/projected/5d066599-937f-44b9-89d0-dadc7d7478e9-kube-api-access-ptbs2\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:02.458858 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:02.458807 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf67s\" (UniqueName: \"kubernetes.io/projected/81d0fc53-fb6b-4437-a486-bb8ab2e3e39a-kube-api-access-qf67s\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:03.059396 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.059350 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" podUID="81d0fc53-fb6b-4437-a486-bb8ab2e3e39a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.180:50052\" within 1s: context deadline exceeded" Oct 30 16:58:03.070989 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.070957 4861 generic.go:356] "Generic (PLEG): container finished" podID="5d066599-937f-44b9-89d0-dadc7d7478e9" containerID="25e082e239af41b617a26f28a0ea7d4213bf497f083660fd530ea406ce4478b2" exitCode=0 Oct 30 16:58:03.071144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.071016 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" Oct 30 16:58:03.071144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.071039 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" event={"ID":"5d066599-937f-44b9-89d0-dadc7d7478e9","Type":"ContainerDied","Data":"25e082e239af41b617a26f28a0ea7d4213bf497f083660fd530ea406ce4478b2"} Oct 30 16:58:03.071144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.071130 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9" event={"ID":"5d066599-937f-44b9-89d0-dadc7d7478e9","Type":"ContainerDied","Data":"e5aa17cb2370dace6250615212edaf2d73ca33bce2b80586442ed3c061a57381"} Oct 30 16:58:03.071271 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.071166 4861 scope.go:122] "RemoveContainer" containerID="25e082e239af41b617a26f28a0ea7d4213bf497f083660fd530ea406ce4478b2" Oct 30 16:58:03.072938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.072918 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" event={"ID":"81d0fc53-fb6b-4437-a486-bb8ab2e3e39a","Type":"ContainerDied","Data":"0c6ae7a15c14b3736b3a4f3efb55a7442cdc26e5e4544402a0923cba2b1c2e84"} Oct 30 16:58:03.073091 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.073076 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m" Oct 30 16:58:03.084254 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.084217 4861 scope.go:122] "RemoveContainer" containerID="25e082e239af41b617a26f28a0ea7d4213bf497f083660fd530ea406ce4478b2" Oct 30 16:58:03.084569 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:58:03.084544 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e082e239af41b617a26f28a0ea7d4213bf497f083660fd530ea406ce4478b2\": container with ID starting with 25e082e239af41b617a26f28a0ea7d4213bf497f083660fd530ea406ce4478b2 not found: ID does not exist" containerID="25e082e239af41b617a26f28a0ea7d4213bf497f083660fd530ea406ce4478b2" Oct 30 16:58:03.084628 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.084587 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e082e239af41b617a26f28a0ea7d4213bf497f083660fd530ea406ce4478b2"} err="failed to get container status \"25e082e239af41b617a26f28a0ea7d4213bf497f083660fd530ea406ce4478b2\": rpc error: code = NotFound desc = could not find container \"25e082e239af41b617a26f28a0ea7d4213bf497f083660fd530ea406ce4478b2\": container with ID starting with 25e082e239af41b617a26f28a0ea7d4213bf497f083660fd530ea406ce4478b2 not found: ID does not exist" Oct 30 16:58:03.084628 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.084613 4861 scope.go:122] "RemoveContainer" containerID="b317c83873850bb81d69281275c4399e9c7db1b64a84c07a59a0ce5959c30a88" Oct 30 16:58:03.127846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.127822 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9"] Oct 30 16:58:03.137896 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.137874 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-77f4f4-646ff5778c-tn8q9"] Oct 30 16:58:03.164155 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.164131 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m"] Oct 30 16:58:03.175326 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.175305 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-77f4f4-774ffb4f4-nk45m"] Oct 30 16:58:03.809002 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.808968 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d066599-937f-44b9-89d0-dadc7d7478e9" path="/var/lib/kubelet/pods/5d066599-937f-44b9-89d0-dadc7d7478e9/volumes" Oct 30 16:58:03.809448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:03.809434 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d0fc53-fb6b-4437-a486-bb8ab2e3e39a" path="/var/lib/kubelet/pods/81d0fc53-fb6b-4437-a486-bb8ab2e3e39a/volumes" Oct 30 16:58:17.289372 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:17.289342 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8"] Oct 30 16:58:17.289897 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:17.289560 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" podUID="f9951912-7c87-4ed5-8525-073105af8c0c" containerName="istio-proxy" containerID="cri-o://e091d2ff1cd52d94d8c061ef7e6fcdafc82a5cd93add9a69a7409407ba52fd93" gracePeriod=30 Oct 30 16:58:17.302755 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:17.302713 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85"] Oct 30 16:58:17.303000 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:17.302952 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" podUID="879aadbb-7e84-4ffc-b078-925a5ff30c43" containerName="istio-proxy" containerID="cri-o://0f6d4f29ba76a33adc3b7c182e44c28072a436f271dc351a8cb18afbc6dd54f5" gracePeriod=30 Oct 30 16:58:17.306327 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:17.306300 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg"] Oct 30 16:58:17.306528 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:17.306505 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" podUID="b22f4e7a-ed39-41e3-9983-253a897bf603" containerName="istio-proxy" containerID="cri-o://801cebc106511cc7c00bdaa0e4a261f6a20ae059f00efa40b7f083ea2a34eb50" gracePeriod=30 Oct 30 16:58:17.314922 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:17.314899 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh"] Oct 30 16:58:17.315122 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:17.315101 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" podUID="e12f55db-ef70-4f34-b3f7-14d431f1ca78" containerName="istio-proxy" containerID="cri-o://a21cbba253f2b833af7f81fb151c5a62dc67d5d6fb737d89c14a2300c40b5531" gracePeriod=30 Oct 30 16:58:19.563821 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:19.563787 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl"] Oct 30 16:58:19.564218 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:19.564020 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" podUID="db2f1614-1fc7-437b-89c8-001c8f180877" containerName="istio-proxy" containerID="cri-o://dfe95bef70a29c1453883c4e11ee85ca50d74f6962b07ac4859707f8cda71ee9" gracePeriod=30 Oct 30 16:58:20.291152 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:58:20.291113 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:58:22.651876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.651846 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:58:22.741756 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.741660 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:58:22.746760 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.746709 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:58:22.752010 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.751958 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:58:22.770308 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770287 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-istio-data\") pod \"f9951912-7c87-4ed5-8525-073105af8c0c\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " Oct 30 16:58:22.770431 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770326 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-istio-envoy\") pod \"f9951912-7c87-4ed5-8525-073105af8c0c\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " Oct 30 16:58:22.770431 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770372 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-podinfo\") pod \"879aadbb-7e84-4ffc-b078-925a5ff30c43\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " Oct 30 16:58:22.770431 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770400 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-envoy\") pod \"879aadbb-7e84-4ffc-b078-925a5ff30c43\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " Oct 30 16:58:22.770619 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770443 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-workload-certs\") pod \"b22f4e7a-ed39-41e3-9983-253a897bf603\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " Oct 30 16:58:22.770619 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770479 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/879aadbb-7e84-4ffc-b078-925a5ff30c43-kube-api-access-27kh4\" (UniqueName: \"kubernetes.io/projected/879aadbb-7e84-4ffc-b078-925a5ff30c43-kube-api-access-27kh4\") pod \"879aadbb-7e84-4ffc-b078-925a5ff30c43\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " Oct 30 16:58:22.770619 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770524 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-workload-certs\") pod \"f9951912-7c87-4ed5-8525-073105af8c0c\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " Oct 30 16:58:22.770619 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770553 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9951912-7c87-4ed5-8525-073105af8c0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9951912-7c87-4ed5-8525-073105af8c0c-istio-podinfo\") pod \"f9951912-7c87-4ed5-8525-073105af8c0c\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " Oct 30 16:58:22.770619 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770604 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-data\") pod \"b22f4e7a-ed39-41e3-9983-253a897bf603\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " Oct 30 16:58:22.770959 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770631 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b22f4e7a-ed39-41e3-9983-253a897bf603-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22f4e7a-ed39-41e3-9983-253a897bf603-istiod-ca-cert\") pod \"b22f4e7a-ed39-41e3-9983-253a897bf603\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " Oct 30 16:58:22.770959 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770667 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9951912-7c87-4ed5-8525-073105af8c0c-kube-api-access-pdpzj\" (UniqueName: \"kubernetes.io/projected/f9951912-7c87-4ed5-8525-073105af8c0c-kube-api-access-pdpzj\") pod \"f9951912-7c87-4ed5-8525-073105af8c0c\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " Oct 30 16:58:22.770959 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770711 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-token\" (UniqueName: \"kubernetes.io/projected/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-token\") pod \"b22f4e7a-ed39-41e3-9983-253a897bf603\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " Oct 30 16:58:22.770959 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770794 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9951912-7c87-4ed5-8525-073105af8c0c-istio-token\" (UniqueName: \"kubernetes.io/projected/f9951912-7c87-4ed5-8525-073105af8c0c-istio-token\") pod \"f9951912-7c87-4ed5-8525-073105af8c0c\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " Oct 30 16:58:22.770959 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770879 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-credential-socket\") pod \"b22f4e7a-ed39-41e3-9983-253a897bf603\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " Oct 30 16:58:22.770959 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770915 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b22f4e7a-ed39-41e3-9983-253a897bf603-kube-api-access-np5kx\" (UniqueName: \"kubernetes.io/projected/b22f4e7a-ed39-41e3-9983-253a897bf603-kube-api-access-np5kx\") pod \"b22f4e7a-ed39-41e3-9983-253a897bf603\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " Oct 30 16:58:22.771374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770954 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-istio-data" pod "f9951912-7c87-4ed5-8525-073105af8c0c" (UID: "f9951912-7c87-4ed5-8525-073105af8c0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.771374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.770970 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-envoy\") pod \"b22f4e7a-ed39-41e3-9983-253a897bf603\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " Oct 30 16:58:22.771374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.771088 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-podinfo\") pod \"b22f4e7a-ed39-41e3-9983-253a897bf603\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " Oct 30 16:58:22.771374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.771131 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-credential-socket\") pod \"f9951912-7c87-4ed5-8525-073105af8c0c\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " Oct 30 16:58:22.771374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.771165 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/879aadbb-7e84-4ffc-b078-925a5ff30c43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/879aadbb-7e84-4ffc-b078-925a5ff30c43-istiod-ca-cert\") pod \"879aadbb-7e84-4ffc-b078-925a5ff30c43\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " Oct 30 16:58:22.771374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.771212 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-workload-socket\") pod \"879aadbb-7e84-4ffc-b078-925a5ff30c43\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " Oct 30 16:58:22.771374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.771249 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9951912-7c87-4ed5-8525-073105af8c0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9951912-7c87-4ed5-8525-073105af8c0c-istiod-ca-cert\") pod \"f9951912-7c87-4ed5-8525-073105af8c0c\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " Oct 30 16:58:22.771374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.771334 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-data" pod "b22f4e7a-ed39-41e3-9983-253a897bf603" (UID: "b22f4e7a-ed39-41e3-9983-253a897bf603"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.771374 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.771354 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-workload-socket\") pod \"b22f4e7a-ed39-41e3-9983-253a897bf603\" (UID: \"b22f4e7a-ed39-41e3-9983-253a897bf603\") " Oct 30 16:58:22.771941 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.771408 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-workload-certs\") pod \"879aadbb-7e84-4ffc-b078-925a5ff30c43\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " Oct 30 16:58:22.771941 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.771693 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-workload-certs" pod "879aadbb-7e84-4ffc-b078-925a5ff30c43" (UID: "879aadbb-7e84-4ffc-b078-925a5ff30c43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.772057 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.772039 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b22f4e7a-ed39-41e3-9983-253a897bf603-istiod-ca-cert" pod "b22f4e7a-ed39-41e3-9983-253a897bf603" (UID: "b22f4e7a-ed39-41e3-9983-253a897bf603"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:58:22.772263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.772245 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-workload-socket" pod "879aadbb-7e84-4ffc-b078-925a5ff30c43" (UID: "879aadbb-7e84-4ffc-b078-925a5ff30c43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.772568 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.772502 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9951912-7c87-4ed5-8525-073105af8c0c-istiod-ca-cert" pod "f9951912-7c87-4ed5-8525-073105af8c0c" (UID: "f9951912-7c87-4ed5-8525-073105af8c0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:58:22.772770 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.772741 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-workload-socket" pod "b22f4e7a-ed39-41e3-9983-253a897bf603" (UID: "b22f4e7a-ed39-41e3-9983-253a897bf603"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.772884 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.772660 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/879aadbb-7e84-4ffc-b078-925a5ff30c43-istiod-ca-cert" pod "879aadbb-7e84-4ffc-b078-925a5ff30c43" (UID: "879aadbb-7e84-4ffc-b078-925a5ff30c43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:58:22.773861 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.772980 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-credential-socket" pod "f9951912-7c87-4ed5-8525-073105af8c0c" (UID: "f9951912-7c87-4ed5-8525-073105af8c0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.773861 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.773339 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-istio-envoy" pod "f9951912-7c87-4ed5-8525-073105af8c0c" (UID: "f9951912-7c87-4ed5-8525-073105af8c0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.773861 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.773594 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-credential-socket" pod "b22f4e7a-ed39-41e3-9983-253a897bf603" (UID: "b22f4e7a-ed39-41e3-9983-253a897bf603"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.774872 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.771712 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-token\" (UniqueName: \"kubernetes.io/projected/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-token\") pod \"879aadbb-7e84-4ffc-b078-925a5ff30c43\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " Oct 30 16:58:22.774974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.774914 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-data\") pod \"879aadbb-7e84-4ffc-b078-925a5ff30c43\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " Oct 30 16:58:22.774974 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.774960 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-credential-socket\") pod \"879aadbb-7e84-4ffc-b078-925a5ff30c43\" (UID: \"879aadbb-7e84-4ffc-b078-925a5ff30c43\") " Oct 30 16:58:22.775126 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.775020 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-workload-socket\") pod \"f9951912-7c87-4ed5-8525-073105af8c0c\" (UID: \"f9951912-7c87-4ed5-8525-073105af8c0c\") " Oct 30 16:58:22.775592 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.775434 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879aadbb-7e84-4ffc-b078-925a5ff30c43-kube-api-access-27kh4" pod "879aadbb-7e84-4ffc-b078-925a5ff30c43" (UID: "879aadbb-7e84-4ffc-b078-925a5ff30c43"). InnerVolumeSpecName "kube-api-access-27kh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:22.775592 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.775472 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9951912-7c87-4ed5-8525-073105af8c0c-istio-token" pod "f9951912-7c87-4ed5-8525-073105af8c0c" (UID: "f9951912-7c87-4ed5-8525-073105af8c0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:22.775774 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.775646 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-workload-socket" pod "f9951912-7c87-4ed5-8525-073105af8c0c" (UID: "f9951912-7c87-4ed5-8525-073105af8c0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.776093 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.776073 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.776183 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.776098 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27kh4\" (UniqueName: \"kubernetes.io/projected/879aadbb-7e84-4ffc-b078-925a5ff30c43-kube-api-access-27kh4\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.776183 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.776113 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.776183 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.776125 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b22f4e7a-ed39-41e3-9983-253a897bf603-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.776183 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.776137 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9951912-7c87-4ed5-8525-073105af8c0c-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.776183 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.776150 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.776183 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.776163 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.776183 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.776176 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/879aadbb-7e84-4ffc-b078-925a5ff30c43-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.776571 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.776189 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.776571 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.776201 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9951912-7c87-4ed5-8525-073105af8c0c-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.776571 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.776215 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.776571 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.776228 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.776571 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.776241 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.776571 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.776253 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.778317 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.777868 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-workload-certs" pod "f9951912-7c87-4ed5-8525-073105af8c0c" (UID: "f9951912-7c87-4ed5-8525-073105af8c0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.778317 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.777974 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-envoy" pod "879aadbb-7e84-4ffc-b078-925a5ff30c43" (UID: "879aadbb-7e84-4ffc-b078-925a5ff30c43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.778317 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.778070 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-data" pod "879aadbb-7e84-4ffc-b078-925a5ff30c43" (UID: "879aadbb-7e84-4ffc-b078-925a5ff30c43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.778317 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.778181 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-workload-certs" pod "b22f4e7a-ed39-41e3-9983-253a897bf603" (UID: "b22f4e7a-ed39-41e3-9983-253a897bf603"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.778317 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.778290 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-credential-socket" pod "879aadbb-7e84-4ffc-b078-925a5ff30c43" (UID: "879aadbb-7e84-4ffc-b078-925a5ff30c43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.778317 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.778303 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-token" pod "b22f4e7a-ed39-41e3-9983-253a897bf603" (UID: "b22f4e7a-ed39-41e3-9983-253a897bf603"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:22.778701 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.778609 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-podinfo" pod "b22f4e7a-ed39-41e3-9983-253a897bf603" (UID: "b22f4e7a-ed39-41e3-9983-253a897bf603"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:58:22.778701 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.778676 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-podinfo" pod "879aadbb-7e84-4ffc-b078-925a5ff30c43" (UID: "879aadbb-7e84-4ffc-b078-925a5ff30c43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:58:22.778848 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.778764 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-envoy" pod "b22f4e7a-ed39-41e3-9983-253a897bf603" (UID: "b22f4e7a-ed39-41e3-9983-253a897bf603"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.778987 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.778886 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-token" pod "879aadbb-7e84-4ffc-b078-925a5ff30c43" (UID: "879aadbb-7e84-4ffc-b078-925a5ff30c43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:22.779900 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.779871 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9951912-7c87-4ed5-8525-073105af8c0c-istio-podinfo" pod "f9951912-7c87-4ed5-8525-073105af8c0c" (UID: "f9951912-7c87-4ed5-8525-073105af8c0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:58:22.780815 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.780790 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b22f4e7a-ed39-41e3-9983-253a897bf603-kube-api-access-np5kx" pod "b22f4e7a-ed39-41e3-9983-253a897bf603" (UID: "b22f4e7a-ed39-41e3-9983-253a897bf603"). InnerVolumeSpecName "kube-api-access-np5kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:22.780904 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.780828 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9951912-7c87-4ed5-8525-073105af8c0c-kube-api-access-pdpzj" pod "f9951912-7c87-4ed5-8525-073105af8c0c" (UID: "f9951912-7c87-4ed5-8525-073105af8c0c"). InnerVolumeSpecName "kube-api-access-pdpzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:22.877715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.877018 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-envoy\") pod \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " Oct 30 16:58:22.877715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.877103 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-workload-certs\") pod \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " Oct 30 16:58:22.877715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.877179 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-token\" (UniqueName: \"kubernetes.io/projected/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-token\") pod \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " Oct 30 16:58:22.877715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.877226 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12f55db-ef70-4f34-b3f7-14d431f1ca78-kube-api-access-vd24w\" (UniqueName: \"kubernetes.io/projected/e12f55db-ef70-4f34-b3f7-14d431f1ca78-kube-api-access-vd24w\") pod \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " Oct 30 16:58:22.877715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.877286 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-podinfo\") pod \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " Oct 30 16:58:22.877715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.877346 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-credential-socket\") pod \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " Oct 30 16:58:22.877715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.877384 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-workload-socket\") pod \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " Oct 30 16:58:22.877715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.877431 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istiod-ca-cert\") pod \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " Oct 30 16:58:22.877715 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.877481 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-data\") pod \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\" (UID: \"e12f55db-ef70-4f34-b3f7-14d431f1ca78\") " Oct 30 16:58:22.879224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.878455 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9951912-7c87-4ed5-8525-073105af8c0c-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.879224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.878483 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9951912-7c87-4ed5-8525-073105af8c0c-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.879224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.878497 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdpzj\" (UniqueName: \"kubernetes.io/projected/f9951912-7c87-4ed5-8525-073105af8c0c-kube-api-access-pdpzj\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.879224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.878514 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.879224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.878531 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np5kx\" (UniqueName: \"kubernetes.io/projected/b22f4e7a-ed39-41e3-9983-253a897bf603-kube-api-access-np5kx\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.879224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.878545 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.879224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.878558 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b22f4e7a-ed39-41e3-9983-253a897bf603-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.879224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.878573 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.879224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.878587 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.879224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.878599 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.879224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.878614 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.879224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.878627 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/879aadbb-7e84-4ffc-b078-925a5ff30c43-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.879224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.878640 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b22f4e7a-ed39-41e3-9983-253a897bf603-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.880655 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.880026 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-workload-certs" pod "e12f55db-ef70-4f34-b3f7-14d431f1ca78" (UID: "e12f55db-ef70-4f34-b3f7-14d431f1ca78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.880655 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.880070 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-data" pod "e12f55db-ef70-4f34-b3f7-14d431f1ca78" (UID: "e12f55db-ef70-4f34-b3f7-14d431f1ca78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.880655 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.880461 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-credential-socket" pod "e12f55db-ef70-4f34-b3f7-14d431f1ca78" (UID: "e12f55db-ef70-4f34-b3f7-14d431f1ca78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.881166 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.880808 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-workload-socket" pod "e12f55db-ef70-4f34-b3f7-14d431f1ca78" (UID: "e12f55db-ef70-4f34-b3f7-14d431f1ca78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.881166 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.880820 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istiod-ca-cert" pod "e12f55db-ef70-4f34-b3f7-14d431f1ca78" (UID: "e12f55db-ef70-4f34-b3f7-14d431f1ca78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:58:22.883060 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.883032 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-token" pod "e12f55db-ef70-4f34-b3f7-14d431f1ca78" (UID: "e12f55db-ef70-4f34-b3f7-14d431f1ca78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:22.883950 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.883924 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-podinfo" pod "e12f55db-ef70-4f34-b3f7-14d431f1ca78" (UID: "e12f55db-ef70-4f34-b3f7-14d431f1ca78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:58:22.884166 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.884129 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12f55db-ef70-4f34-b3f7-14d431f1ca78-kube-api-access-vd24w" pod "e12f55db-ef70-4f34-b3f7-14d431f1ca78" (UID: "e12f55db-ef70-4f34-b3f7-14d431f1ca78"). InnerVolumeSpecName "kube-api-access-vd24w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:22.885018 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.885000 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-envoy" pod "e12f55db-ef70-4f34-b3f7-14d431f1ca78" (UID: "e12f55db-ef70-4f34-b3f7-14d431f1ca78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:22.980413 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.980385 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.980413 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.980412 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.980604 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.980421 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.980604 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.980430 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.980604 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.980444 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.980604 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.980455 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12f55db-ef70-4f34-b3f7-14d431f1ca78-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.980604 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.980463 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.980604 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.980471 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd24w\" (UniqueName: \"kubernetes.io/projected/e12f55db-ef70-4f34-b3f7-14d431f1ca78-kube-api-access-vd24w\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:22.980604 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:22.980478 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12f55db-ef70-4f34-b3f7-14d431f1ca78-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:23.288286 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.288247 4861 generic.go:356] "Generic (PLEG): container finished" podID="b22f4e7a-ed39-41e3-9983-253a897bf603" containerID="801cebc106511cc7c00bdaa0e4a261f6a20ae059f00efa40b7f083ea2a34eb50" exitCode=0 Oct 30 16:58:23.288468 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.288299 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" Oct 30 16:58:23.288468 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.288339 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" event={"ID":"b22f4e7a-ed39-41e3-9983-253a897bf603","Type":"ContainerDied","Data":"801cebc106511cc7c00bdaa0e4a261f6a20ae059f00efa40b7f083ea2a34eb50"} Oct 30 16:58:23.288468 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.288398 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg" event={"ID":"b22f4e7a-ed39-41e3-9983-253a897bf603","Type":"ContainerDied","Data":"e10d9eacded89726d9b813005d0e58945a03f015f036e642770668628c4cf328"} Oct 30 16:58:23.288468 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.288422 4861 scope.go:122] "RemoveContainer" containerID="801cebc106511cc7c00bdaa0e4a261f6a20ae059f00efa40b7f083ea2a34eb50" Oct 30 16:58:23.290114 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.290087 4861 generic.go:356] "Generic (PLEG): container finished" podID="e12f55db-ef70-4f34-b3f7-14d431f1ca78" containerID="a21cbba253f2b833af7f81fb151c5a62dc67d5d6fb737d89c14a2300c40b5531" exitCode=0 Oct 30 16:58:23.290240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.290189 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" Oct 30 16:58:23.290240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.290189 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" event={"ID":"e12f55db-ef70-4f34-b3f7-14d431f1ca78","Type":"ContainerDied","Data":"a21cbba253f2b833af7f81fb151c5a62dc67d5d6fb737d89c14a2300c40b5531"} Oct 30 16:58:23.290364 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.290263 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh" event={"ID":"e12f55db-ef70-4f34-b3f7-14d431f1ca78","Type":"ContainerDied","Data":"5fdee313cc7a8855a172e80e57fcc9bb551a318c92467e6f0ba8a19c307bc50b"} Oct 30 16:58:23.292003 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.291972 4861 generic.go:356] "Generic (PLEG): container finished" podID="879aadbb-7e84-4ffc-b078-925a5ff30c43" containerID="0f6d4f29ba76a33adc3b7c182e44c28072a436f271dc351a8cb18afbc6dd54f5" exitCode=0 Oct 30 16:58:23.292263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.292019 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" Oct 30 16:58:23.292263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.292073 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" event={"ID":"879aadbb-7e84-4ffc-b078-925a5ff30c43","Type":"ContainerDied","Data":"0f6d4f29ba76a33adc3b7c182e44c28072a436f271dc351a8cb18afbc6dd54f5"} Oct 30 16:58:23.292263 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.292112 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85" event={"ID":"879aadbb-7e84-4ffc-b078-925a5ff30c43","Type":"ContainerDied","Data":"060b8a46c456e61cba17721627ac7c4cf88b217d9a9998d82885181e69acddb7"} Oct 30 16:58:23.293873 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.293793 4861 generic.go:356] "Generic (PLEG): container finished" podID="f9951912-7c87-4ed5-8525-073105af8c0c" containerID="e091d2ff1cd52d94d8c061ef7e6fcdafc82a5cd93add9a69a7409407ba52fd93" exitCode=0 Oct 30 16:58:23.293873 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.293853 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" Oct 30 16:58:23.294054 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.293896 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" event={"ID":"f9951912-7c87-4ed5-8525-073105af8c0c","Type":"ContainerDied","Data":"e091d2ff1cd52d94d8c061ef7e6fcdafc82a5cd93add9a69a7409407ba52fd93"} Oct 30 16:58:23.294054 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.293939 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8" event={"ID":"f9951912-7c87-4ed5-8525-073105af8c0c","Type":"ContainerDied","Data":"c614e377637a96fb05f68429fb66da708fb48873a380ac52294fd19b69791e37"} Oct 30 16:58:23.308080 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.308055 4861 scope.go:122] "RemoveContainer" containerID="801cebc106511cc7c00bdaa0e4a261f6a20ae059f00efa40b7f083ea2a34eb50" Oct 30 16:58:23.308444 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:58:23.308422 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"801cebc106511cc7c00bdaa0e4a261f6a20ae059f00efa40b7f083ea2a34eb50\": container with ID starting with 801cebc106511cc7c00bdaa0e4a261f6a20ae059f00efa40b7f083ea2a34eb50 not found: ID does not exist" containerID="801cebc106511cc7c00bdaa0e4a261f6a20ae059f00efa40b7f083ea2a34eb50" Oct 30 16:58:23.308650 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.308451 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"801cebc106511cc7c00bdaa0e4a261f6a20ae059f00efa40b7f083ea2a34eb50"} err="failed to get container status \"801cebc106511cc7c00bdaa0e4a261f6a20ae059f00efa40b7f083ea2a34eb50\": rpc error: code = NotFound desc = could not find container \"801cebc106511cc7c00bdaa0e4a261f6a20ae059f00efa40b7f083ea2a34eb50\": container with ID starting with 801cebc106511cc7c00bdaa0e4a261f6a20ae059f00efa40b7f083ea2a34eb50 not found: ID does not exist" Oct 30 16:58:23.308650 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.308472 4861 scope.go:122] "RemoveContainer" containerID="a21cbba253f2b833af7f81fb151c5a62dc67d5d6fb737d89c14a2300c40b5531" Oct 30 16:58:23.351140 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.351123 4861 scope.go:122] "RemoveContainer" containerID="a21cbba253f2b833af7f81fb151c5a62dc67d5d6fb737d89c14a2300c40b5531" Oct 30 16:58:23.351404 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:58:23.351384 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a21cbba253f2b833af7f81fb151c5a62dc67d5d6fb737d89c14a2300c40b5531\": container with ID starting with a21cbba253f2b833af7f81fb151c5a62dc67d5d6fb737d89c14a2300c40b5531 not found: ID does not exist" containerID="a21cbba253f2b833af7f81fb151c5a62dc67d5d6fb737d89c14a2300c40b5531" Oct 30 16:58:23.351469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.351413 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21cbba253f2b833af7f81fb151c5a62dc67d5d6fb737d89c14a2300c40b5531"} err="failed to get container status \"a21cbba253f2b833af7f81fb151c5a62dc67d5d6fb737d89c14a2300c40b5531\": rpc error: code = NotFound desc = could not find container \"a21cbba253f2b833af7f81fb151c5a62dc67d5d6fb737d89c14a2300c40b5531\": container with ID starting with a21cbba253f2b833af7f81fb151c5a62dc67d5d6fb737d89c14a2300c40b5531 not found: ID does not exist" Oct 30 16:58:23.351469 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.351429 4861 scope.go:122] "RemoveContainer" containerID="0f6d4f29ba76a33adc3b7c182e44c28072a436f271dc351a8cb18afbc6dd54f5" Oct 30 16:58:23.362282 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.362266 4861 scope.go:122] "RemoveContainer" containerID="0f6d4f29ba76a33adc3b7c182e44c28072a436f271dc351a8cb18afbc6dd54f5" Oct 30 16:58:23.362537 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:58:23.362516 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f6d4f29ba76a33adc3b7c182e44c28072a436f271dc351a8cb18afbc6dd54f5\": container with ID starting with 0f6d4f29ba76a33adc3b7c182e44c28072a436f271dc351a8cb18afbc6dd54f5 not found: ID does not exist" containerID="0f6d4f29ba76a33adc3b7c182e44c28072a436f271dc351a8cb18afbc6dd54f5" Oct 30 16:58:23.362610 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.362544 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f6d4f29ba76a33adc3b7c182e44c28072a436f271dc351a8cb18afbc6dd54f5"} err="failed to get container status \"0f6d4f29ba76a33adc3b7c182e44c28072a436f271dc351a8cb18afbc6dd54f5\": rpc error: code = NotFound desc = could not find container \"0f6d4f29ba76a33adc3b7c182e44c28072a436f271dc351a8cb18afbc6dd54f5\": container with ID starting with 0f6d4f29ba76a33adc3b7c182e44c28072a436f271dc351a8cb18afbc6dd54f5 not found: ID does not exist" Oct 30 16:58:23.362610 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.362560 4861 scope.go:122] "RemoveContainer" containerID="e091d2ff1cd52d94d8c061ef7e6fcdafc82a5cd93add9a69a7409407ba52fd93" Oct 30 16:58:23.373034 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.373019 4861 scope.go:122] "RemoveContainer" containerID="e091d2ff1cd52d94d8c061ef7e6fcdafc82a5cd93add9a69a7409407ba52fd93" Oct 30 16:58:23.373286 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:58:23.373268 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e091d2ff1cd52d94d8c061ef7e6fcdafc82a5cd93add9a69a7409407ba52fd93\": container with ID starting with e091d2ff1cd52d94d8c061ef7e6fcdafc82a5cd93add9a69a7409407ba52fd93 not found: ID does not exist" containerID="e091d2ff1cd52d94d8c061ef7e6fcdafc82a5cd93add9a69a7409407ba52fd93" Oct 30 16:58:23.373342 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.373295 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e091d2ff1cd52d94d8c061ef7e6fcdafc82a5cd93add9a69a7409407ba52fd93"} err="failed to get container status \"e091d2ff1cd52d94d8c061ef7e6fcdafc82a5cd93add9a69a7409407ba52fd93\": rpc error: code = NotFound desc = could not find container \"e091d2ff1cd52d94d8c061ef7e6fcdafc82a5cd93add9a69a7409407ba52fd93\": container with ID starting with e091d2ff1cd52d94d8c061ef7e6fcdafc82a5cd93add9a69a7409407ba52fd93 not found: ID does not exist" Oct 30 16:58:23.393608 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.393587 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85"] Oct 30 16:58:23.406866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.406839 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-pjf85"] Oct 30 16:58:23.442590 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.442562 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh"] Oct 30 16:58:23.446398 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.446376 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-5phdh"] Oct 30 16:58:23.481357 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.481328 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8"] Oct 30 16:58:23.485144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.485124 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-fv4w8"] Oct 30 16:58:23.521568 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.521546 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg"] Oct 30 16:58:23.528313 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.528136 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-kcnsg"] Oct 30 16:58:23.808482 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.808449 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="879aadbb-7e84-4ffc-b078-925a5ff30c43" path="/var/lib/kubelet/pods/879aadbb-7e84-4ffc-b078-925a5ff30c43/volumes" Oct 30 16:58:23.808994 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.808977 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b22f4e7a-ed39-41e3-9983-253a897bf603" path="/var/lib/kubelet/pods/b22f4e7a-ed39-41e3-9983-253a897bf603/volumes" Oct 30 16:58:23.809419 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.809405 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12f55db-ef70-4f34-b3f7-14d431f1ca78" path="/var/lib/kubelet/pods/e12f55db-ef70-4f34-b3f7-14d431f1ca78/volumes" Oct 30 16:58:23.810310 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:23.810283 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9951912-7c87-4ed5-8525-073105af8c0c" path="/var/lib/kubelet/pods/f9951912-7c87-4ed5-8525-073105af8c0c/volumes" Oct 30 16:58:24.840957 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.840896 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:58:24.904087 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.904054 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db2f1614-1fc7-437b-89c8-001c8f180877-istio-token\" (UniqueName: \"kubernetes.io/projected/db2f1614-1fc7-437b-89c8-001c8f180877-istio-token\") pod \"db2f1614-1fc7-437b-89c8-001c8f180877\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " Oct 30 16:58:24.904238 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.904139 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-credential-socket\") pod \"db2f1614-1fc7-437b-89c8-001c8f180877\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " Oct 30 16:58:24.904238 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.904186 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-workload-certs\") pod \"db2f1614-1fc7-437b-89c8-001c8f180877\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " Oct 30 16:58:24.904382 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.904287 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db2f1614-1fc7-437b-89c8-001c8f180877-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2f1614-1fc7-437b-89c8-001c8f180877-istio-podinfo\") pod \"db2f1614-1fc7-437b-89c8-001c8f180877\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " Oct 30 16:58:24.904382 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.904323 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-istio-envoy\") pod \"db2f1614-1fc7-437b-89c8-001c8f180877\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " Oct 30 16:58:24.904493 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.904384 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db2f1614-1fc7-437b-89c8-001c8f180877-kube-api-access-9rt25\" (UniqueName: \"kubernetes.io/projected/db2f1614-1fc7-437b-89c8-001c8f180877-kube-api-access-9rt25\") pod \"db2f1614-1fc7-437b-89c8-001c8f180877\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " Oct 30 16:58:24.904493 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.904479 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-workload-socket\") pod \"db2f1614-1fc7-437b-89c8-001c8f180877\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " Oct 30 16:58:24.904602 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.904542 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db2f1614-1fc7-437b-89c8-001c8f180877-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2f1614-1fc7-437b-89c8-001c8f180877-istiod-ca-cert\") pod \"db2f1614-1fc7-437b-89c8-001c8f180877\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " Oct 30 16:58:24.904658 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.904630 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-istio-data\") pod \"db2f1614-1fc7-437b-89c8-001c8f180877\" (UID: \"db2f1614-1fc7-437b-89c8-001c8f180877\") " Oct 30 16:58:24.904712 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.904656 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-workload-certs" pod "db2f1614-1fc7-437b-89c8-001c8f180877" (UID: "db2f1614-1fc7-437b-89c8-001c8f180877"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:24.905085 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.904905 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-credential-socket" pod "db2f1614-1fc7-437b-89c8-001c8f180877" (UID: "db2f1614-1fc7-437b-89c8-001c8f180877"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:24.905600 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.905287 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-workload-socket" pod "db2f1614-1fc7-437b-89c8-001c8f180877" (UID: "db2f1614-1fc7-437b-89c8-001c8f180877"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:24.905600 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.905331 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:24.905600 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.905353 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:24.905600 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.905553 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-istio-data" pod "db2f1614-1fc7-437b-89c8-001c8f180877" (UID: "db2f1614-1fc7-437b-89c8-001c8f180877"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:24.905931 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.905593 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db2f1614-1fc7-437b-89c8-001c8f180877-istiod-ca-cert" pod "db2f1614-1fc7-437b-89c8-001c8f180877" (UID: "db2f1614-1fc7-437b-89c8-001c8f180877"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:58:24.907332 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.907300 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db2f1614-1fc7-437b-89c8-001c8f180877-istio-podinfo" pod "db2f1614-1fc7-437b-89c8-001c8f180877" (UID: "db2f1614-1fc7-437b-89c8-001c8f180877"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:58:24.907459 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.907403 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2f1614-1fc7-437b-89c8-001c8f180877-istio-token" pod "db2f1614-1fc7-437b-89c8-001c8f180877" (UID: "db2f1614-1fc7-437b-89c8-001c8f180877"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:24.907574 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.907557 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-istio-envoy" pod "db2f1614-1fc7-437b-89c8-001c8f180877" (UID: "db2f1614-1fc7-437b-89c8-001c8f180877"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:24.908162 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:24.908141 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2f1614-1fc7-437b-89c8-001c8f180877-kube-api-access-9rt25" pod "db2f1614-1fc7-437b-89c8-001c8f180877" (UID: "db2f1614-1fc7-437b-89c8-001c8f180877"). InnerVolumeSpecName "kube-api-access-9rt25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:25.007562 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.007513 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db2f1614-1fc7-437b-89c8-001c8f180877-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:25.007562 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.007554 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:25.007885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.007569 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rt25\" (UniqueName: \"kubernetes.io/projected/db2f1614-1fc7-437b-89c8-001c8f180877-kube-api-access-9rt25\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:25.007885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.007587 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:25.007885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.007602 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db2f1614-1fc7-437b-89c8-001c8f180877-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:25.007885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.007617 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db2f1614-1fc7-437b-89c8-001c8f180877-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:25.007885 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.007630 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db2f1614-1fc7-437b-89c8-001c8f180877-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:58:25.314136 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.314098 4861 generic.go:356] "Generic (PLEG): container finished" podID="db2f1614-1fc7-437b-89c8-001c8f180877" containerID="dfe95bef70a29c1453883c4e11ee85ca50d74f6962b07ac4859707f8cda71ee9" exitCode=0 Oct 30 16:58:25.314308 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.314153 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" Oct 30 16:58:25.314308 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.314194 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" event={"ID":"db2f1614-1fc7-437b-89c8-001c8f180877","Type":"ContainerDied","Data":"dfe95bef70a29c1453883c4e11ee85ca50d74f6962b07ac4859707f8cda71ee9"} Oct 30 16:58:25.314308 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.314257 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl" event={"ID":"db2f1614-1fc7-437b-89c8-001c8f180877","Type":"ContainerDied","Data":"f973f4c2ecd3ef985b5ab0cde119f50431ed09d16aa5bff70dc99c06c2ac608c"} Oct 30 16:58:25.314308 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.314292 4861 scope.go:122] "RemoveContainer" containerID="dfe95bef70a29c1453883c4e11ee85ca50d74f6962b07ac4859707f8cda71ee9" Oct 30 16:58:25.326425 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.326408 4861 scope.go:122] "RemoveContainer" containerID="898a23a78e9f5fdabab7bee16d8bbc1ce72c5364ba30b91a0c4a64de0430918c" Oct 30 16:58:25.335967 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.335944 4861 scope.go:122] "RemoveContainer" containerID="dfe95bef70a29c1453883c4e11ee85ca50d74f6962b07ac4859707f8cda71ee9" Oct 30 16:58:25.336229 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:58:25.336211 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe95bef70a29c1453883c4e11ee85ca50d74f6962b07ac4859707f8cda71ee9\": container with ID starting with dfe95bef70a29c1453883c4e11ee85ca50d74f6962b07ac4859707f8cda71ee9 not found: ID does not exist" containerID="dfe95bef70a29c1453883c4e11ee85ca50d74f6962b07ac4859707f8cda71ee9" Oct 30 16:58:25.336289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.336239 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe95bef70a29c1453883c4e11ee85ca50d74f6962b07ac4859707f8cda71ee9"} err="failed to get container status \"dfe95bef70a29c1453883c4e11ee85ca50d74f6962b07ac4859707f8cda71ee9\": rpc error: code = NotFound desc = could not find container \"dfe95bef70a29c1453883c4e11ee85ca50d74f6962b07ac4859707f8cda71ee9\": container with ID starting with dfe95bef70a29c1453883c4e11ee85ca50d74f6962b07ac4859707f8cda71ee9 not found: ID does not exist" Oct 30 16:58:25.336289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.336255 4861 scope.go:122] "RemoveContainer" containerID="898a23a78e9f5fdabab7bee16d8bbc1ce72c5364ba30b91a0c4a64de0430918c" Oct 30 16:58:25.336528 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:58:25.336503 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"898a23a78e9f5fdabab7bee16d8bbc1ce72c5364ba30b91a0c4a64de0430918c\": container with ID starting with 898a23a78e9f5fdabab7bee16d8bbc1ce72c5364ba30b91a0c4a64de0430918c not found: ID does not exist" containerID="898a23a78e9f5fdabab7bee16d8bbc1ce72c5364ba30b91a0c4a64de0430918c" Oct 30 16:58:25.336575 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.336536 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"898a23a78e9f5fdabab7bee16d8bbc1ce72c5364ba30b91a0c4a64de0430918c"} err="failed to get container status \"898a23a78e9f5fdabab7bee16d8bbc1ce72c5364ba30b91a0c4a64de0430918c\": rpc error: code = NotFound desc = could not find container \"898a23a78e9f5fdabab7bee16d8bbc1ce72c5364ba30b91a0c4a64de0430918c\": container with ID starting with 898a23a78e9f5fdabab7bee16d8bbc1ce72c5364ba30b91a0c4a64de0430918c not found: ID does not exist" Oct 30 16:58:25.377286 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.377257 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl"] Oct 30 16:58:25.388416 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.388393 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-77f4f4-openshift-default-6d49f8894f-rx2cl"] Oct 30 16:58:25.809376 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:25.809345 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db2f1614-1fc7-437b-89c8-001c8f180877" path="/var/lib/kubelet/pods/db2f1614-1fc7-437b-89c8-001c8f180877/volumes" Oct 30 16:58:26.364187 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.364157 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj"] Oct 30 16:58:26.365103 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365074 4861 cpu_manager.go:414] "containerMap: removing container" podUID="f9951912-7c87-4ed5-8525-073105af8c0c" containerName="istio-proxy" Oct 30 16:58:26.365103 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365102 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9951912-7c87-4ed5-8525-073105af8c0c" containerName="istio-proxy" Oct 30 16:58:26.365311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365143 4861 cpu_manager.go:414] "containerMap: removing container" podUID="81d0fc53-fb6b-4437-a486-bb8ab2e3e39a" containerName="payload-processing" Oct 30 16:58:26.365311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365152 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d0fc53-fb6b-4437-a486-bb8ab2e3e39a" containerName="payload-processing" Oct 30 16:58:26.365311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365176 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b22f4e7a-ed39-41e3-9983-253a897bf603" containerName="istio-proxy" Oct 30 16:58:26.365311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365183 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b22f4e7a-ed39-41e3-9983-253a897bf603" containerName="istio-proxy" Oct 30 16:58:26.365311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365217 4861 cpu_manager.go:414] "containerMap: removing container" podUID="db2f1614-1fc7-437b-89c8-001c8f180877" containerName="istio-proxy" Oct 30 16:58:26.365311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365225 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2f1614-1fc7-437b-89c8-001c8f180877" containerName="istio-proxy" Oct 30 16:58:26.365311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365245 4861 cpu_manager.go:414] "containerMap: removing container" podUID="e12f55db-ef70-4f34-b3f7-14d431f1ca78" containerName="istio-proxy" Oct 30 16:58:26.365311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365253 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12f55db-ef70-4f34-b3f7-14d431f1ca78" containerName="istio-proxy" Oct 30 16:58:26.365311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365265 4861 cpu_manager.go:414] "containerMap: removing container" podUID="db2f1614-1fc7-437b-89c8-001c8f180877" containerName="istio-proxy" Oct 30 16:58:26.365311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365272 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db2f1614-1fc7-437b-89c8-001c8f180877" containerName="istio-proxy" Oct 30 16:58:26.365311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365282 4861 cpu_manager.go:414] "containerMap: removing container" podUID="5d066599-937f-44b9-89d0-dadc7d7478e9" containerName="payload-pre-processing" Oct 30 16:58:26.365311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365289 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d066599-937f-44b9-89d0-dadc7d7478e9" containerName="payload-pre-processing" Oct 30 16:58:26.365311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365306 4861 cpu_manager.go:414] "containerMap: removing container" podUID="879aadbb-7e84-4ffc-b078-925a5ff30c43" containerName="istio-proxy" Oct 30 16:58:26.365311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365313 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="879aadbb-7e84-4ffc-b078-925a5ff30c43" containerName="istio-proxy" Oct 30 16:58:26.365878 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365470 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9951912-7c87-4ed5-8525-073105af8c0c" containerName="istio-proxy" Oct 30 16:58:26.365878 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365488 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d0fc53-fb6b-4437-a486-bb8ab2e3e39a" containerName="payload-processing" Oct 30 16:58:26.365878 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365501 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b22f4e7a-ed39-41e3-9983-253a897bf603" containerName="istio-proxy" Oct 30 16:58:26.365878 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365515 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="db2f1614-1fc7-437b-89c8-001c8f180877" containerName="istio-proxy" Oct 30 16:58:26.365878 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365527 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12f55db-ef70-4f34-b3f7-14d431f1ca78" containerName="istio-proxy" Oct 30 16:58:26.365878 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365534 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="db2f1614-1fc7-437b-89c8-001c8f180877" containerName="istio-proxy" Oct 30 16:58:26.365878 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365544 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d066599-937f-44b9-89d0-dadc7d7478e9" containerName="payload-pre-processing" Oct 30 16:58:26.365878 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.365555 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="879aadbb-7e84-4ffc-b078-925a5ff30c43" containerName="istio-proxy" Oct 30 16:58:26.371299 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.371275 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.395480 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.395459 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj"] Oct 30 16:58:26.410034 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.409979 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-18fb02-openshift-default-dockercfg-5xthx\"" Oct 30 16:58:26.425205 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.425177 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-workload-certs\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.425340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.425306 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-token\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.425385 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.425358 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-envoy\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.425424 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.425412 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4187598-e0e1-4787-87ad-d7eb1f0af775-istiod-ca-cert\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.425470 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.425454 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-data\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.425551 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.425524 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-workload-socket\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.425617 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.425603 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-podinfo\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.425682 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.425668 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-credential-socket\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.425825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.425806 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdbhg\" (UniqueName: \"kubernetes.io/projected/a4187598-e0e1-4787-87ad-d7eb1f0af775-kube-api-access-xdbhg\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.526978 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.526930 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdbhg\" (UniqueName: \"kubernetes.io/projected/a4187598-e0e1-4787-87ad-d7eb1f0af775-kube-api-access-xdbhg\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.527188 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.527002 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-workload-certs\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.527188 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.527071 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-token\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.527188 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.527105 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-envoy\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.527188 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.527145 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4187598-e0e1-4787-87ad-d7eb1f0af775-istiod-ca-cert\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.527188 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.527182 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-data\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.527495 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.527248 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-workload-socket\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.527495 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.527323 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-podinfo\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.527495 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.527386 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-credential-socket\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.527495 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.527400 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-workload-certs\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.527714 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.527565 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-data\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.527797 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.527709 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-workload-socket\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.527890 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.527871 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-credential-socket\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.527949 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.527917 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4187598-e0e1-4787-87ad-d7eb1f0af775-istiod-ca-cert\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.529452 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.529427 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-envoy\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.530006 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.529984 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-podinfo\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.557657 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.557627 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdbhg\" (UniqueName: \"kubernetes.io/projected/a4187598-e0e1-4787-87ad-d7eb1f0af775-kube-api-access-xdbhg\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.567786 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.565473 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-token\") pod \"e2e-explicit-18fb02-openshift-default-547576b45-rwrnj\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:26.702009 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:26.698662 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:27.160833 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:27.160808 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj"] Oct 30 16:58:27.161635 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:58:27.161607 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4187598_e0e1_4787_87ad_d7eb1f0af775.slice/crio-81d10c23c3f1d981dc31d768984c720de8bc0ca03f05e98bf22182367f088596 WatchSource:0}: Error finding container 81d10c23c3f1d981dc31d768984c720de8bc0ca03f05e98bf22182367f088596: Status 404 returned error can't find the container with id 81d10c23c3f1d981dc31d768984c720de8bc0ca03f05e98bf22182367f088596 Oct 30 16:58:27.164486 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:27.164454 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:58:27.164598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:27.164530 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:58:27.164598 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:27.164575 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:58:27.332846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:27.332762 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" event={"ID":"a4187598-e0e1-4787-87ad-d7eb1f0af775","Type":"ContainerStarted","Data":"feb4db1f6bc55c5eec3c96521dd7f30045d129b682620b41a0a79152a823dfe4"} Oct 30 16:58:27.332846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:27.332834 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" event={"ID":"a4187598-e0e1-4787-87ad-d7eb1f0af775","Type":"ContainerStarted","Data":"81d10c23c3f1d981dc31d768984c720de8bc0ca03f05e98bf22182367f088596"} Oct 30 16:58:27.385220 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:27.385157 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" podStartSLOduration=1.385138048 podStartE2EDuration="1.385138048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:27.379788104 +0000 UTC m=+2137.722462142" watchObservedRunningTime="2026-10-30 16:58:27.385138048 +0000 UTC m=+2137.727812087" Oct 30 16:58:27.699860 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:27.699832 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:28.706331 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:28.706300 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:29.346950 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:29.346920 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:29.347788 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:29.347764 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:58:31.685513 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:31.685473 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5"] Oct 30 16:58:31.690512 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:31.690487 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" Oct 30 16:58:31.709624 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:31.709597 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5"] Oct 30 16:58:31.718224 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:31.718196 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-18fb02\"" Oct 30 16:58:31.718529 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:31.718510 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-18fb02-dockercfg-8698g\"" Oct 30 16:58:31.907748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:31.890514 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c84db179-8068-4c19-be7e-228a37b89a2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5\" (UID: \"c84db179-8068-4c19-be7e-228a37b89a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" Oct 30 16:58:31.907748 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:31.890703 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn8zd\" (UniqueName: \"kubernetes.io/projected/c84db179-8068-4c19-be7e-228a37b89a2c-kube-api-access-pn8zd\") pod \"payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5\" (UID: \"c84db179-8068-4c19-be7e-228a37b89a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" Oct 30 16:58:31.964846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:31.964769 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z"] Oct 30 16:58:31.970269 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:31.970235 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z"] Oct 30 16:58:31.970409 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:31.970356 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" Oct 30 16:58:31.992365 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:31.992339 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c84db179-8068-4c19-be7e-228a37b89a2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5\" (UID: \"c84db179-8068-4c19-be7e-228a37b89a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" Oct 30 16:58:31.992630 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:31.992613 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn8zd\" (UniqueName: \"kubernetes.io/projected/c84db179-8068-4c19-be7e-228a37b89a2c-kube-api-access-pn8zd\") pod \"payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5\" (UID: \"c84db179-8068-4c19-be7e-228a37b89a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" Oct 30 16:58:31.993108 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:31.993091 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c84db179-8068-4c19-be7e-228a37b89a2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5\" (UID: \"c84db179-8068-4c19-be7e-228a37b89a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" Oct 30 16:58:32.017043 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:32.017010 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn8zd\" (UniqueName: \"kubernetes.io/projected/c84db179-8068-4c19-be7e-228a37b89a2c-kube-api-access-pn8zd\") pod \"payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5\" (UID: \"c84db179-8068-4c19-be7e-228a37b89a2c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" Oct 30 16:58:32.093863 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:32.093835 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06557e2c-fcca-4155-8322-5b666975f8be-plugins-config-volume\") pod \"payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z\" (UID: \"06557e2c-fcca-4155-8322-5b666975f8be\") " pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" Oct 30 16:58:32.093988 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:32.093954 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxwjz\" (UniqueName: \"kubernetes.io/projected/06557e2c-fcca-4155-8322-5b666975f8be-kube-api-access-rxwjz\") pod \"payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z\" (UID: \"06557e2c-fcca-4155-8322-5b666975f8be\") " pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" Oct 30 16:58:32.194781 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:32.194662 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06557e2c-fcca-4155-8322-5b666975f8be-plugins-config-volume\") pod \"payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z\" (UID: \"06557e2c-fcca-4155-8322-5b666975f8be\") " pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" Oct 30 16:58:32.194963 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:32.194932 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxwjz\" (UniqueName: \"kubernetes.io/projected/06557e2c-fcca-4155-8322-5b666975f8be-kube-api-access-rxwjz\") pod \"payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z\" (UID: \"06557e2c-fcca-4155-8322-5b666975f8be\") " pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" Oct 30 16:58:32.195339 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:32.195316 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06557e2c-fcca-4155-8322-5b666975f8be-plugins-config-volume\") pod \"payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z\" (UID: \"06557e2c-fcca-4155-8322-5b666975f8be\") " pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" Oct 30 16:58:32.223021 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:32.222984 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxwjz\" (UniqueName: \"kubernetes.io/projected/06557e2c-fcca-4155-8322-5b666975f8be-kube-api-access-rxwjz\") pod \"payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z\" (UID: \"06557e2c-fcca-4155-8322-5b666975f8be\") " pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" Oct 30 16:58:32.286143 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:32.286113 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" Oct 30 16:58:32.306015 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:32.305991 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" Oct 30 16:58:32.649212 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:32.648523 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z"] Oct 30 16:58:32.656913 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:32.656888 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5"] Oct 30 16:58:32.658902 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:58:32.658863 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84db179_8068_4c19_be7e_228a37b89a2c.slice/crio-acb9103008a9cf1019c96306c261fb273c8a4649dc12fc2ab61eb0dcce8833a5 WatchSource:0}: Error finding container acb9103008a9cf1019c96306c261fb273c8a4649dc12fc2ab61eb0dcce8833a5: Status 404 returned error can't find the container with id acb9103008a9cf1019c96306c261fb273c8a4649dc12fc2ab61eb0dcce8833a5 Oct 30 16:58:33.385107 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:33.385072 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" event={"ID":"c84db179-8068-4c19-be7e-228a37b89a2c","Type":"ContainerStarted","Data":"87adfd3480e86814e07a781bc667ed9adc8d036f84c1a65ae4c39d2e75f374b0"} Oct 30 16:58:33.385576 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:33.385151 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" Oct 30 16:58:33.385576 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:33.385177 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" event={"ID":"c84db179-8068-4c19-be7e-228a37b89a2c","Type":"ContainerStarted","Data":"acb9103008a9cf1019c96306c261fb273c8a4649dc12fc2ab61eb0dcce8833a5"} Oct 30 16:58:33.386682 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:33.386656 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" event={"ID":"06557e2c-fcca-4155-8322-5b666975f8be","Type":"ContainerStarted","Data":"3f462b06ffdb8da5af4bdcb5611b31f544778d90cd51485f6721589f267c8e1b"} Oct 30 16:58:33.386846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:33.386704 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" Oct 30 16:58:33.386846 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:33.386745 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" event={"ID":"06557e2c-fcca-4155-8322-5b666975f8be","Type":"ContainerStarted","Data":"54fa6b3d8344c4a25b15e21b3d9af37f2da4e751c70ae3e5cd5f380ead7cfed2"} Oct 30 16:58:33.432980 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:33.432932 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" podStartSLOduration=2.4329175530000002 podStartE2EDuration="2.432917553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:33.430058215 +0000 UTC m=+2143.772732257" watchObservedRunningTime="2026-10-30 16:58:33.432917553 +0000 UTC m=+2143.775591590" Oct 30 16:58:33.473637 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:33.473593 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" podStartSLOduration=2.473580119 podStartE2EDuration="2.473580119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:33.471366981 +0000 UTC m=+2143.814041034" watchObservedRunningTime="2026-10-30 16:58:33.473580119 +0000 UTC m=+2143.816254153" Oct 30 16:58:44.397067 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:44.396987 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" Oct 30 16:58:44.397907 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:44.397881 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" Oct 30 16:58:50.290553 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:58:50.290516 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:58:51.091778 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:58:51.091718 4861 scope.go:122] "RemoveContainer" containerID="68b7249d6114c332b25b1148e3da19b0920dd8cfc835a5c520d0e8e334a943ad" Oct 30 16:59:01.363533 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:01.363495 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4"] Oct 30 16:59:01.364134 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:01.363711 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" podUID="310bed42-3cfb-4b4d-b075-de011a27909f" containerName="istio-proxy" containerID="cri-o://c80ddde3acd7e2e7fd3fe54cd825da9aaaa133b7c426de6cf250e9b77cb67ea6" gracePeriod=30 Oct 30 16:59:01.378036 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:01.378006 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh"] Oct 30 16:59:01.378304 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:01.378262 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" podUID="84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" containerName="istio-proxy" containerID="cri-o://119215ef6d621ae288697b0bda15e01342d36c9ff5ec0d4704f23f5e9a66d5a9" gracePeriod=30 Oct 30 16:59:01.381394 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:01.381365 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj"] Oct 30 16:59:01.381555 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:01.381531 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" podUID="021fd3dd-e215-430d-b702-b01f2e90053e" containerName="istio-proxy" containerID="cri-o://cffd064b2e612cdae4a41fd21e97c67057c190b3d0536dccc84b45046f0ba084" gracePeriod=30 Oct 30 16:59:01.389551 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:01.389526 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s"] Oct 30 16:59:01.389777 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:01.389753 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" podUID="16535ec9-31d5-4620-b9c9-2367ba14e26d" containerName="istio-proxy" containerID="cri-o://15893985a97ed68896f6e550193012a9e14d1f9824477bfa644f09595a11642f" gracePeriod=30 Oct 30 16:59:04.443749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:04.438655 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:59:04.443749 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:04.438798 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" podUID="021fd3dd-e215-430d-b702-b01f2e90053e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:59:06.677488 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.676780 4861 generic.go:356] "Generic (PLEG): container finished" podID="84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" containerID="119215ef6d621ae288697b0bda15e01342d36c9ff5ec0d4704f23f5e9a66d5a9" exitCode=0 Oct 30 16:59:06.677488 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.676873 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" event={"ID":"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8","Type":"ContainerDied","Data":"119215ef6d621ae288697b0bda15e01342d36c9ff5ec0d4704f23f5e9a66d5a9"} Oct 30 16:59:06.680363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.679181 4861 generic.go:356] "Generic (PLEG): container finished" podID="310bed42-3cfb-4b4d-b075-de011a27909f" containerID="c80ddde3acd7e2e7fd3fe54cd825da9aaaa133b7c426de6cf250e9b77cb67ea6" exitCode=0 Oct 30 16:59:06.680363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.679332 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" event={"ID":"310bed42-3cfb-4b4d-b075-de011a27909f","Type":"ContainerDied","Data":"c80ddde3acd7e2e7fd3fe54cd825da9aaaa133b7c426de6cf250e9b77cb67ea6"} Oct 30 16:59:06.680363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.679372 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" event={"ID":"310bed42-3cfb-4b4d-b075-de011a27909f","Type":"ContainerDied","Data":"2bcb22d5261c562f13a71d9f99ca425103a99725185816c5b5de9661c665b799"} Oct 30 16:59:06.680363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.679395 4861 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bcb22d5261c562f13a71d9f99ca425103a99725185816c5b5de9661c665b799" Oct 30 16:59:06.682867 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.682472 4861 generic.go:356] "Generic (PLEG): container finished" podID="16535ec9-31d5-4620-b9c9-2367ba14e26d" containerID="15893985a97ed68896f6e550193012a9e14d1f9824477bfa644f09595a11642f" exitCode=0 Oct 30 16:59:06.682867 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.682555 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" event={"ID":"16535ec9-31d5-4620-b9c9-2367ba14e26d","Type":"ContainerDied","Data":"15893985a97ed68896f6e550193012a9e14d1f9824477bfa644f09595a11642f"} Oct 30 16:59:06.684866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.684616 4861 generic.go:356] "Generic (PLEG): container finished" podID="021fd3dd-e215-430d-b702-b01f2e90053e" containerID="cffd064b2e612cdae4a41fd21e97c67057c190b3d0536dccc84b45046f0ba084" exitCode=0 Oct 30 16:59:06.684866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.684677 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" event={"ID":"021fd3dd-e215-430d-b702-b01f2e90053e","Type":"ContainerDied","Data":"cffd064b2e612cdae4a41fd21e97c67057c190b3d0536dccc84b45046f0ba084"} Oct 30 16:59:06.710824 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.710798 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:59:06.764647 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.764619 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:59:06.767704 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.767675 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:59:06.771785 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.771767 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:59:06.800352 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800317 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-workload-socket\") pod \"021fd3dd-e215-430d-b702-b01f2e90053e\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " Oct 30 16:59:06.800352 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800356 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/310bed42-3cfb-4b4d-b075-de011a27909f-istio-token\" (UniqueName: \"kubernetes.io/projected/310bed42-3cfb-4b4d-b075-de011a27909f-istio-token\") pod \"310bed42-3cfb-4b4d-b075-de011a27909f\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " Oct 30 16:59:06.800603 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800380 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-data\") pod \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " Oct 30 16:59:06.800603 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800428 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-podinfo\") pod \"16535ec9-31d5-4620-b9c9-2367ba14e26d\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " Oct 30 16:59:06.800603 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800454 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-envoy\") pod \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " Oct 30 16:59:06.800603 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800486 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-token\" (UniqueName: \"kubernetes.io/projected/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-token\") pod \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " Oct 30 16:59:06.800603 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800522 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-workload-certs\") pod \"310bed42-3cfb-4b4d-b075-de011a27909f\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " Oct 30 16:59:06.800603 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800561 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-data\") pod \"16535ec9-31d5-4620-b9c9-2367ba14e26d\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " Oct 30 16:59:06.800603 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800600 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-credential-socket\") pod \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " Oct 30 16:59:06.801035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800631 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-credential-socket\") pod \"16535ec9-31d5-4620-b9c9-2367ba14e26d\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " Oct 30 16:59:06.801035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800659 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-podinfo\") pod \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " Oct 30 16:59:06.801035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800685 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-data" pod "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" (UID: "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.801035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800696 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/310bed42-3cfb-4b4d-b075-de011a27909f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310bed42-3cfb-4b4d-b075-de011a27909f-istio-podinfo\") pod \"310bed42-3cfb-4b4d-b075-de011a27909f\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " Oct 30 16:59:06.801035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800757 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-istio-envoy\") pod \"021fd3dd-e215-430d-b702-b01f2e90053e\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " Oct 30 16:59:06.801035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800788 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-istio-data\") pod \"021fd3dd-e215-430d-b702-b01f2e90053e\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " Oct 30 16:59:06.801035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800822 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/021fd3dd-e215-430d-b702-b01f2e90053e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021fd3dd-e215-430d-b702-b01f2e90053e-istio-podinfo\") pod \"021fd3dd-e215-430d-b702-b01f2e90053e\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " Oct 30 16:59:06.801035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800873 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16535ec9-31d5-4620-b9c9-2367ba14e26d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16535ec9-31d5-4620-b9c9-2367ba14e26d-istiod-ca-cert\") pod \"16535ec9-31d5-4620-b9c9-2367ba14e26d\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " Oct 30 16:59:06.801035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800908 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/310bed42-3cfb-4b4d-b075-de011a27909f-kube-api-access-lwddr\" (UniqueName: \"kubernetes.io/projected/310bed42-3cfb-4b4d-b075-de011a27909f-kube-api-access-lwddr\") pod \"310bed42-3cfb-4b4d-b075-de011a27909f\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " Oct 30 16:59:06.801035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800921 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-credential-socket" pod "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" (UID: "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.801035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.800948 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-workload-certs\") pod \"021fd3dd-e215-430d-b702-b01f2e90053e\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " Oct 30 16:59:06.801035 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801019 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/310bed42-3cfb-4b4d-b075-de011a27909f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310bed42-3cfb-4b4d-b075-de011a27909f-istiod-ca-cert\") pod \"310bed42-3cfb-4b4d-b075-de011a27909f\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801055 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-credential-socket\") pod \"310bed42-3cfb-4b4d-b075-de011a27909f\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801093 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-envoy\") pod \"16535ec9-31d5-4620-b9c9-2367ba14e26d\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801129 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-workload-socket\") pod \"16535ec9-31d5-4620-b9c9-2367ba14e26d\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801137 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-workload-certs" pod "310bed42-3cfb-4b4d-b075-de011a27909f" (UID: "310bed42-3cfb-4b4d-b075-de011a27909f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801162 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-data" pod "16535ec9-31d5-4620-b9c9-2367ba14e26d" (UID: "16535ec9-31d5-4620-b9c9-2367ba14e26d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801165 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-kube-api-access-x9bf5\" (UniqueName: \"kubernetes.io/projected/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-kube-api-access-x9bf5\") pod \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801212 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021fd3dd-e215-430d-b702-b01f2e90053e-istio-token\" (UniqueName: \"kubernetes.io/projected/021fd3dd-e215-430d-b702-b01f2e90053e-istio-token\") pod \"021fd3dd-e215-430d-b702-b01f2e90053e\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801250 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-istio-envoy\") pod \"310bed42-3cfb-4b4d-b075-de011a27909f\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801292 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istiod-ca-cert\") pod \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801332 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021fd3dd-e215-430d-b702-b01f2e90053e-kube-api-access-qfkrr\" (UniqueName: \"kubernetes.io/projected/021fd3dd-e215-430d-b702-b01f2e90053e-kube-api-access-qfkrr\") pod \"021fd3dd-e215-430d-b702-b01f2e90053e\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801368 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-workload-certs\") pod \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801406 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-istio-data\") pod \"310bed42-3cfb-4b4d-b075-de011a27909f\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801456 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310bed42-3cfb-4b4d-b075-de011a27909f-istiod-ca-cert" pod "310bed42-3cfb-4b4d-b075-de011a27909f" (UID: "310bed42-3cfb-4b4d-b075-de011a27909f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801468 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-workload-socket\") pod \"310bed42-3cfb-4b4d-b075-de011a27909f\" (UID: \"310bed42-3cfb-4b4d-b075-de011a27909f\") " Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801541 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-workload-certs\") pod \"16535ec9-31d5-4620-b9c9-2367ba14e26d\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801557 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16535ec9-31d5-4620-b9c9-2367ba14e26d-istiod-ca-cert" pod "16535ec9-31d5-4620-b9c9-2367ba14e26d" (UID: "16535ec9-31d5-4620-b9c9-2367ba14e26d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801581 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-credential-socket\") pod \"021fd3dd-e215-430d-b702-b01f2e90053e\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801618 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-workload-socket\") pod \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\" (UID: \"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8\") " Oct 30 16:59:06.801676 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801649 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/021fd3dd-e215-430d-b702-b01f2e90053e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021fd3dd-e215-430d-b702-b01f2e90053e-istiod-ca-cert\") pod \"021fd3dd-e215-430d-b702-b01f2e90053e\" (UID: \"021fd3dd-e215-430d-b702-b01f2e90053e\") " Oct 30 16:59:06.802696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801693 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-token\" (UniqueName: \"kubernetes.io/projected/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-token\") pod \"16535ec9-31d5-4620-b9c9-2367ba14e26d\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " Oct 30 16:59:06.802696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801757 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16535ec9-31d5-4620-b9c9-2367ba14e26d-kube-api-access-hxh6r\" (UniqueName: \"kubernetes.io/projected/16535ec9-31d5-4620-b9c9-2367ba14e26d-kube-api-access-hxh6r\") pod \"16535ec9-31d5-4620-b9c9-2367ba14e26d\" (UID: \"16535ec9-31d5-4620-b9c9-2367ba14e26d\") " Oct 30 16:59:06.802696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.801883 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-credential-socket" pod "16535ec9-31d5-4620-b9c9-2367ba14e26d" (UID: "16535ec9-31d5-4620-b9c9-2367ba14e26d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.802696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.802415 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.802696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.802439 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.802696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.802454 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.802696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.802468 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.802696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.802483 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.802696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.802498 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16535ec9-31d5-4620-b9c9-2367ba14e26d-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.802696 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.802512 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/310bed42-3cfb-4b4d-b075-de011a27909f-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.803340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.802847 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-workload-socket" pod "021fd3dd-e215-430d-b702-b01f2e90053e" (UID: "021fd3dd-e215-430d-b702-b01f2e90053e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.803688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.803623 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-workload-certs" pod "021fd3dd-e215-430d-b702-b01f2e90053e" (UID: "021fd3dd-e215-430d-b702-b01f2e90053e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.803953 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.803844 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-workload-certs" pod "16535ec9-31d5-4620-b9c9-2367ba14e26d" (UID: "16535ec9-31d5-4620-b9c9-2367ba14e26d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.804047 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.804026 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-workload-certs" pod "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" (UID: "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.804534 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.804221 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-istio-data" pod "310bed42-3cfb-4b4d-b075-de011a27909f" (UID: "310bed42-3cfb-4b4d-b075-de011a27909f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.804534 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.804285 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310bed42-3cfb-4b4d-b075-de011a27909f-kube-api-access-lwddr" pod "310bed42-3cfb-4b4d-b075-de011a27909f" (UID: "310bed42-3cfb-4b4d-b075-de011a27909f"). InnerVolumeSpecName "kube-api-access-lwddr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:06.804534 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.804435 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-credential-socket" pod "021fd3dd-e215-430d-b702-b01f2e90053e" (UID: "021fd3dd-e215-430d-b702-b01f2e90053e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.804534 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.804440 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-workload-socket" pod "310bed42-3cfb-4b4d-b075-de011a27909f" (UID: "310bed42-3cfb-4b4d-b075-de011a27909f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.804890 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.804662 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-workload-socket" pod "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" (UID: "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.805289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.805088 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-workload-socket" pod "16535ec9-31d5-4620-b9c9-2367ba14e26d" (UID: "16535ec9-31d5-4620-b9c9-2367ba14e26d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.805289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.805116 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021fd3dd-e215-430d-b702-b01f2e90053e-istiod-ca-cert" pod "021fd3dd-e215-430d-b702-b01f2e90053e" (UID: "021fd3dd-e215-430d-b702-b01f2e90053e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:06.805866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.805567 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istiod-ca-cert" pod "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" (UID: "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:06.805866 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.805573 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16535ec9-31d5-4620-b9c9-2367ba14e26d-kube-api-access-hxh6r" pod "16535ec9-31d5-4620-b9c9-2367ba14e26d" (UID: "16535ec9-31d5-4620-b9c9-2367ba14e26d"). InnerVolumeSpecName "kube-api-access-hxh6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:06.806103 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.806017 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-credential-socket" pod "310bed42-3cfb-4b4d-b075-de011a27909f" (UID: "310bed42-3cfb-4b4d-b075-de011a27909f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.808329 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.808304 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-kube-api-access-x9bf5" pod "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" (UID: "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8"). InnerVolumeSpecName "kube-api-access-x9bf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:06.808525 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.808339 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-podinfo" pod "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" (UID: "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:59:06.808640 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.808408 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310bed42-3cfb-4b4d-b075-de011a27909f-istio-token" pod "310bed42-3cfb-4b4d-b075-de011a27909f" (UID: "310bed42-3cfb-4b4d-b075-de011a27909f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:06.808782 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.808476 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-istio-data" pod "021fd3dd-e215-430d-b702-b01f2e90053e" (UID: "021fd3dd-e215-430d-b702-b01f2e90053e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.809466 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.809431 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-token" pod "16535ec9-31d5-4620-b9c9-2367ba14e26d" (UID: "16535ec9-31d5-4620-b9c9-2367ba14e26d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:06.809562 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.809491 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/021fd3dd-e215-430d-b702-b01f2e90053e-istio-podinfo" pod "021fd3dd-e215-430d-b702-b01f2e90053e" (UID: "021fd3dd-e215-430d-b702-b01f2e90053e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:59:06.809624 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.809591 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-istio-envoy" pod "310bed42-3cfb-4b4d-b075-de011a27909f" (UID: "310bed42-3cfb-4b4d-b075-de011a27909f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.809624 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.809611 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021fd3dd-e215-430d-b702-b01f2e90053e-istio-token" pod "021fd3dd-e215-430d-b702-b01f2e90053e" (UID: "021fd3dd-e215-430d-b702-b01f2e90053e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:06.809624 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.809610 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/310bed42-3cfb-4b4d-b075-de011a27909f-istio-podinfo" pod "310bed42-3cfb-4b4d-b075-de011a27909f" (UID: "310bed42-3cfb-4b4d-b075-de011a27909f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:59:06.809830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.809649 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021fd3dd-e215-430d-b702-b01f2e90053e-kube-api-access-qfkrr" pod "021fd3dd-e215-430d-b702-b01f2e90053e" (UID: "021fd3dd-e215-430d-b702-b01f2e90053e"). InnerVolumeSpecName "kube-api-access-qfkrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:06.809830 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.809764 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-token" pod "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" (UID: "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:06.809941 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.809846 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-envoy" pod "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" (UID: "84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.810135 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.810106 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-podinfo" pod "16535ec9-31d5-4620-b9c9-2367ba14e26d" (UID: "16535ec9-31d5-4620-b9c9-2367ba14e26d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:59:06.810449 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.810426 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-envoy" pod "16535ec9-31d5-4620-b9c9-2367ba14e26d" (UID: "16535ec9-31d5-4620-b9c9-2367ba14e26d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.811240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.811220 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-istio-envoy" pod "021fd3dd-e215-430d-b702-b01f2e90053e" (UID: "021fd3dd-e215-430d-b702-b01f2e90053e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:06.903966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.903880 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.903966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.903915 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.903966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.903931 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.903966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.903949 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9bf5\" (UniqueName: \"kubernetes.io/projected/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-kube-api-access-x9bf5\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.903966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.903963 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/021fd3dd-e215-430d-b702-b01f2e90053e-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.903977 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.903990 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904001 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfkrr\" (UniqueName: \"kubernetes.io/projected/021fd3dd-e215-430d-b702-b01f2e90053e-kube-api-access-qfkrr\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904026 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904039 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904051 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/310bed42-3cfb-4b4d-b075-de011a27909f-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904064 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16535ec9-31d5-4620-b9c9-2367ba14e26d-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904077 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904089 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904104 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/021fd3dd-e215-430d-b702-b01f2e90053e-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904119 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904132 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxh6r\" (UniqueName: \"kubernetes.io/projected/16535ec9-31d5-4620-b9c9-2367ba14e26d-kube-api-access-hxh6r\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904147 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904160 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/310bed42-3cfb-4b4d-b075-de011a27909f-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904174 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16535ec9-31d5-4620-b9c9-2367ba14e26d-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904186 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904198 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904210 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904222 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/310bed42-3cfb-4b4d-b075-de011a27909f-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904235 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904249 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904261 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/021fd3dd-e215-430d-b702-b01f2e90053e-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904273 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwddr\" (UniqueName: \"kubernetes.io/projected/310bed42-3cfb-4b4d-b075-de011a27909f-kube-api-access-lwddr\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:06.904276 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:06.904286 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/021fd3dd-e215-430d-b702-b01f2e90053e-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:07.693240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.693162 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" Oct 30 16:59:07.693240 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.693189 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj" event={"ID":"021fd3dd-e215-430d-b702-b01f2e90053e","Type":"ContainerDied","Data":"4cc1398fe45554ace108a26ee975ce2f1af51a39cde59d3ac694894bfdf88069"} Oct 30 16:59:07.693743 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.693246 4861 scope.go:122] "RemoveContainer" containerID="cffd064b2e612cdae4a41fd21e97c67057c190b3d0536dccc84b45046f0ba084" Oct 30 16:59:07.694965 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.694947 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" Oct 30 16:59:07.695116 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.695051 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh" event={"ID":"84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8","Type":"ContainerDied","Data":"88b41789fdab356b487e452c3c14f9879bcbe1f5823453e3ccf1caceaf8e46ba"} Oct 30 16:59:07.696662 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.696640 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4" Oct 30 16:59:07.697810 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.697790 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" event={"ID":"16535ec9-31d5-4620-b9c9-2367ba14e26d","Type":"ContainerDied","Data":"954fc773ef988a28aece4b1c24d2c0677128e4fb2eed70d71535fad47a40f079"} Oct 30 16:59:07.697968 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.697831 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s" Oct 30 16:59:07.706865 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.706848 4861 scope.go:122] "RemoveContainer" containerID="119215ef6d621ae288697b0bda15e01342d36c9ff5ec0d4704f23f5e9a66d5a9" Oct 30 16:59:07.719030 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.719010 4861 scope.go:122] "RemoveContainer" containerID="15893985a97ed68896f6e550193012a9e14d1f9824477bfa644f09595a11642f" Oct 30 16:59:07.797084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.797057 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4"] Oct 30 16:59:07.822066 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.822041 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-nsgb4"] Oct 30 16:59:07.839596 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.839564 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh"] Oct 30 16:59:07.843092 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.843070 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-kwkqh"] Oct 30 16:59:07.881416 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.881390 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s"] Oct 30 16:59:07.885044 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.885015 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-clf5s"] Oct 30 16:59:07.920535 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.920506 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj"] Oct 30 16:59:07.924890 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:07.924870 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-79jsj"] Oct 30 16:59:09.810364 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:09.810334 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="021fd3dd-e215-430d-b702-b01f2e90053e" path="/var/lib/kubelet/pods/021fd3dd-e215-430d-b702-b01f2e90053e/volumes" Oct 30 16:59:09.810826 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:09.810809 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16535ec9-31d5-4620-b9c9-2367ba14e26d" path="/var/lib/kubelet/pods/16535ec9-31d5-4620-b9c9-2367ba14e26d/volumes" Oct 30 16:59:09.811250 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:09.811237 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="310bed42-3cfb-4b4d-b075-de011a27909f" path="/var/lib/kubelet/pods/310bed42-3cfb-4b4d-b075-de011a27909f/volumes" Oct 30 16:59:09.812045 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:09.812022 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" path="/var/lib/kubelet/pods/84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8/volumes" Oct 30 16:59:18.750744 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.750248 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf"] Oct 30 16:59:18.751213 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.751182 4861 cpu_manager.go:414] "containerMap: removing container" podUID="16535ec9-31d5-4620-b9c9-2367ba14e26d" containerName="istio-proxy" Oct 30 16:59:18.751213 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.751201 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16535ec9-31d5-4620-b9c9-2367ba14e26d" containerName="istio-proxy" Oct 30 16:59:18.751311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.751219 4861 cpu_manager.go:414] "containerMap: removing container" podUID="310bed42-3cfb-4b4d-b075-de011a27909f" containerName="istio-proxy" Oct 30 16:59:18.751311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.751226 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="310bed42-3cfb-4b4d-b075-de011a27909f" containerName="istio-proxy" Oct 30 16:59:18.751311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.751244 4861 cpu_manager.go:414] "containerMap: removing container" podUID="84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" containerName="istio-proxy" Oct 30 16:59:18.751311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.751251 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" containerName="istio-proxy" Oct 30 16:59:18.751311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.751302 4861 cpu_manager.go:414] "containerMap: removing container" podUID="021fd3dd-e215-430d-b702-b01f2e90053e" containerName="istio-proxy" Oct 30 16:59:18.751311 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.751308 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021fd3dd-e215-430d-b702-b01f2e90053e" containerName="istio-proxy" Oct 30 16:59:18.751532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.751467 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="16535ec9-31d5-4620-b9c9-2367ba14e26d" containerName="istio-proxy" Oct 30 16:59:18.751532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.751480 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="310bed42-3cfb-4b4d-b075-de011a27909f" containerName="istio-proxy" Oct 30 16:59:18.751532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.751493 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a6ea1d-68b5-4225-a5ac-d6e8fc3b19c8" containerName="istio-proxy" Oct 30 16:59:18.751532 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.751511 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="021fd3dd-e215-430d-b702-b01f2e90053e" containerName="istio-proxy" Oct 30 16:59:18.755982 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.755956 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.775134 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.775107 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf"] Oct 30 16:59:18.785168 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.785140 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2035f6\"/\"kube-root-ca.crt\"" Oct 30 16:59:18.785296 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.785206 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-2035f6\"/\"openshift-service-ca.crt\"" Oct 30 16:59:18.785618 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.785586 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2035f6\"/\"noten-model-2035f6-kserve-self-signed-certs\"" Oct 30 16:59:18.785700 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.785689 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-2035f6\"/\"default-dockercfg-xg726\"" Oct 30 16:59:18.808736 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.808689 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-model-cache\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.808938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.808743 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30c1072-479f-430a-b0e7-6739582a58cc-tls-certs\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.808938 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.808871 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-dshm\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.809060 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.808957 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-tmp-dir\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.809111 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.809038 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxn92\" (UniqueName: \"kubernetes.io/projected/b30c1072-479f-430a-b0e7-6739582a58cc-kube-api-access-xxn92\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.809360 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.809337 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-home\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.910924 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.910877 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxn92\" (UniqueName: \"kubernetes.io/projected/b30c1072-479f-430a-b0e7-6739582a58cc-kube-api-access-xxn92\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.911153 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.911071 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-home\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.911254 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.911238 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-model-cache\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.911340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.911267 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30c1072-479f-430a-b0e7-6739582a58cc-tls-certs\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.911340 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.911325 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-dshm\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.911458 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.911432 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-tmp-dir\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.911600 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.911572 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-home\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.911853 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.911827 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-tmp-dir\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.911981 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.911961 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-model-cache\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.914376 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.914352 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-dshm\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.914653 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.914636 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30c1072-479f-430a-b0e7-6739582a58cc-tls-certs\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:18.935566 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:18.935536 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxn92\" (UniqueName: \"kubernetes.io/projected/b30c1072-479f-430a-b0e7-6739582a58cc-kube-api-access-xxn92\") pod \"noten-model-2035f6-kserve-6667df986f-hjlbf\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:19.077019 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:19.076938 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:19.400121 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:19.400095 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf"] Oct 30 16:59:19.401204 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:59:19.401177 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb30c1072_479f_430a_b0e7_6739582a58cc.slice/crio-35d19b281807981fc6cca0bc2f3a43e997e97b1595412d0fe36bb63269136add WatchSource:0}: Error finding container 35d19b281807981fc6cca0bc2f3a43e997e97b1595412d0fe36bb63269136add: Status 404 returned error can't find the container with id 35d19b281807981fc6cca0bc2f3a43e997e97b1595412d0fe36bb63269136add Oct 30 16:59:19.795446 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:19.795409 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" event={"ID":"b30c1072-479f-430a-b0e7-6739582a58cc","Type":"ContainerStarted","Data":"cefd959068a104a7a378bd62b56ce1578c96e23e49b094c2de28dfa769445b9f"} Oct 30 16:59:19.795875 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:19.795454 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" event={"ID":"b30c1072-479f-430a-b0e7-6739582a58cc","Type":"ContainerStarted","Data":"35d19b281807981fc6cca0bc2f3a43e997e97b1595412d0fe36bb63269136add"} Oct 30 16:59:19.839165 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:19.839111 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" podStartSLOduration=1.83909478 podStartE2EDuration="1.83909478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:19.8379221 +0000 UTC m=+2190.180596140" watchObservedRunningTime="2026-10-30 16:59:19.83909478 +0000 UTC m=+2190.181768823" Oct 30 16:59:20.077425 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:20.077345 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:20.289802 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:59:20.289758 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:59:21.811853 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:21.811816 4861 generic.go:356] "Generic (PLEG): container finished" podID="a4187598-e0e1-4787-87ad-d7eb1f0af775" containerID="feb4db1f6bc55c5eec3c96521dd7f30045d129b682620b41a0a79152a823dfe4" exitCode=0 Oct 30 16:59:21.812323 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:21.811855 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" event={"ID":"a4187598-e0e1-4787-87ad-d7eb1f0af775","Type":"ContainerDied","Data":"feb4db1f6bc55c5eec3c96521dd7f30045d129b682620b41a0a79152a823dfe4"} Oct 30 16:59:21.812323 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:21.812269 4861 scope.go:122] "RemoveContainer" containerID="feb4db1f6bc55c5eec3c96521dd7f30045d129b682620b41a0a79152a823dfe4" Oct 30 16:59:21.813815 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:21.813791 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:59:21.813882 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:21.813855 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:59:21.813919 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:21.813887 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 16:59:22.841909 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:22.841867 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" event={"ID":"a4187598-e0e1-4787-87ad-d7eb1f0af775","Type":"ContainerStarted","Data":"fda01eadaf6dd2e13b17245450a5a39e4114799d97eac30fc6ba5d0777ee3669"} Oct 30 16:59:23.699793 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:23.699760 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:59:23.699993 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:23.699918 4861 patch_prober.go:29] interesting pod/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 16:59:23.699993 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:23.699969 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" podUID="a4187598-e0e1-4787-87ad-d7eb1f0af775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 16:59:24.699957 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:24.699921 4861 patch_prober.go:29] interesting pod/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 16:59:24.700381 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:24.699985 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" podUID="a4187598-e0e1-4787-87ad-d7eb1f0af775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 16:59:25.699509 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:25.699470 4861 patch_prober.go:29] interesting pod/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 16:59:25.699683 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:25.699522 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" podUID="a4187598-e0e1-4787-87ad-d7eb1f0af775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 16:59:26.699465 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:26.699432 4861 patch_prober.go:29] interesting pod/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 16:59:26.699903 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:26.699500 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" podUID="a4187598-e0e1-4787-87ad-d7eb1f0af775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 16:59:27.700100 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:27.700066 4861 patch_prober.go:29] interesting pod/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" start-of-body= Oct 30 16:59:27.700494 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:27.700118 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" podUID="a4187598-e0e1-4787-87ad-d7eb1f0af775" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.203:15021/healthz/ready\": dial tcp 10.132.0.203:15021: connect: connection refused" Oct 30 16:59:28.705026 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:28.704988 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:59:28.705464 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:28.705447 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:59:28.706338 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:28.706318 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 16:59:29.077316 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:29.077283 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:29.089753 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:29.089701 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:29.910227 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:29.910196 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:42.116765 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:42.116544 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf"] Oct 30 16:59:43.589641 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.589606 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75"] Oct 30 16:59:43.596225 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.596193 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.615564 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.615531 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75"] Oct 30 16:59:43.622557 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.622530 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-18fb02\"/\"kube-root-ca.crt\"" Oct 30 16:59:43.622876 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.622856 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-18fb02\"/\"openshift-service-ca.crt\"" Oct 30 16:59:43.673570 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.673539 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-18fb02\"/\"default-dockercfg-h4m65\"" Oct 30 16:59:43.673759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.673605 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-18fb02\"/\"explicit-model-18fb02-kserve-self-signed-certs\"" Oct 30 16:59:43.701473 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.701440 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-tmp-dir\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.701759 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.701703 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-home\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.701874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.701795 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-model-cache\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.701874 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.701848 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb7ae54-914d-4f58-a85f-e360f8082832-tls-certs\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.701991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.701901 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbd9m\" (UniqueName: \"kubernetes.io/projected/eeb7ae54-914d-4f58-a85f-e360f8082832-kube-api-access-cbd9m\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.701991 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.701959 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-dshm\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.803498 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.803458 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-home\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.803655 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.803542 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-model-cache\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.803655 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.803566 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb7ae54-914d-4f58-a85f-e360f8082832-tls-certs\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.803655 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.803593 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbd9m\" (UniqueName: \"kubernetes.io/projected/eeb7ae54-914d-4f58-a85f-e360f8082832-kube-api-access-cbd9m\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.803655 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.803644 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-dshm\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.803956 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.803760 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-tmp-dir\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.803956 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.803947 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-home\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.804061 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.803950 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-model-cache\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.804262 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.804238 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-tmp-dir\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.806447 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.806425 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-dshm\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.806917 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.806896 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb7ae54-914d-4f58-a85f-e360f8082832-tls-certs\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.853608 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.853489 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbd9m\" (UniqueName: \"kubernetes.io/projected/eeb7ae54-914d-4f58-a85f-e360f8082832-kube-api-access-cbd9m\") pod \"explicit-model-18fb02-kserve-84758fffbf-bjl75\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:43.918250 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:43.917019 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:44.244966 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:44.244935 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75"] Oct 30 16:59:44.246292 ip-10-0-131-179 kubenswrapper[4861]: W1030 16:59:44.246257 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeeb7ae54_914d_4f58_a85f_e360f8082832.slice/crio-c5278a95d314b225f2fd7aca1b789984ac2283fb6efbe76244828278c3d3681a WatchSource:0}: Error finding container c5278a95d314b225f2fd7aca1b789984ac2283fb6efbe76244828278c3d3681a: Status 404 returned error can't find the container with id c5278a95d314b225f2fd7aca1b789984ac2283fb6efbe76244828278c3d3681a Oct 30 16:59:45.003757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:45.003682 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" event={"ID":"eeb7ae54-914d-4f58-a85f-e360f8082832","Type":"ContainerStarted","Data":"e97fd15d3f4f7c53736b583e83ec111b79ce000269d3f0456d4a894a61ab8863"} Oct 30 16:59:45.003757 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:45.003750 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" event={"ID":"eeb7ae54-914d-4f58-a85f-e360f8082832","Type":"ContainerStarted","Data":"c5278a95d314b225f2fd7aca1b789984ac2283fb6efbe76244828278c3d3681a"} Oct 30 16:59:45.048144 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:45.048093 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" podStartSLOduration=2.048081139 podStartE2EDuration="2.048081139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:45.045586936 +0000 UTC m=+2215.388260971" watchObservedRunningTime="2026-10-30 16:59:45.048081139 +0000 UTC m=+2215.390755178" Oct 30 16:59:45.919909 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:45.919877 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:50.291531 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:59:50.291486 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 16:59:51.154289 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:51.154249 4861 scope.go:122] "RemoveContainer" containerID="c80ddde3acd7e2e7fd3fe54cd825da9aaaa133b7c426de6cf250e9b77cb67ea6" Oct 30 16:59:52.053908 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:52.053865 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn"] Oct 30 16:59:53.919073 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:53.919033 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:53.932797 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:53.932768 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:54.130182 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:54.130150 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 16:59:57.143336 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.143298 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" podUID="b30c1072-479f-430a-b0e7-6739582a58cc" containerName="main" containerID="cri-o://cefd959068a104a7a378bd62b56ce1578c96e23e49b094c2de28dfa769445b9f" gracePeriod=45 Oct 30 16:59:57.433106 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.433078 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:57.454268 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.454245 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-tmp-dir\") pod \"b30c1072-479f-430a-b0e7-6739582a58cc\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " Oct 30 16:59:57.454387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.454347 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-home\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-home\") pod \"b30c1072-479f-430a-b0e7-6739582a58cc\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " Oct 30 16:59:57.454387 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.454377 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b30c1072-479f-430a-b0e7-6739582a58cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/b30c1072-479f-430a-b0e7-6739582a58cc-tls-certs\") pod \"b30c1072-479f-430a-b0e7-6739582a58cc\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " Oct 30 16:59:57.454522 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.454399 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30c1072-479f-430a-b0e7-6739582a58cc-kube-api-access-xxn92\" (UniqueName: \"kubernetes.io/projected/b30c1072-479f-430a-b0e7-6739582a58cc-kube-api-access-xxn92\") pod \"b30c1072-479f-430a-b0e7-6739582a58cc\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " Oct 30 16:59:57.454522 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.454486 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-tmp-dir" pod "b30c1072-479f-430a-b0e7-6739582a58cc" (UID: "b30c1072-479f-430a-b0e7-6739582a58cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:57.454522 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.454506 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-dshm\") pod \"b30c1072-479f-430a-b0e7-6739582a58cc\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " Oct 30 16:59:57.454688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.454599 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-model-cache\") pod \"b30c1072-479f-430a-b0e7-6739582a58cc\" (UID: \"b30c1072-479f-430a-b0e7-6739582a58cc\") " Oct 30 16:59:57.454688 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.454631 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-home" pod "b30c1072-479f-430a-b0e7-6739582a58cc" (UID: "b30c1072-479f-430a-b0e7-6739582a58cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:57.454914 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.454887 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-model-cache" pod "b30c1072-479f-430a-b0e7-6739582a58cc" (UID: "b30c1072-479f-430a-b0e7-6739582a58cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:57.455430 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.455408 4861 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:57.455527 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.455436 4861 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:57.455527 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.455451 4861 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:57.456968 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.456937 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b30c1072-479f-430a-b0e7-6739582a58cc-tls-certs" pod "b30c1072-479f-430a-b0e7-6739582a58cc" (UID: "b30c1072-479f-430a-b0e7-6739582a58cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:57.457084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.457050 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30c1072-479f-430a-b0e7-6739582a58cc-kube-api-access-xxn92" pod "b30c1072-479f-430a-b0e7-6739582a58cc" (UID: "b30c1072-479f-430a-b0e7-6739582a58cc"). InnerVolumeSpecName "kube-api-access-xxn92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:57.457142 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.457120 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-dshm" pod "b30c1072-479f-430a-b0e7-6739582a58cc" (UID: "b30c1072-479f-430a-b0e7-6739582a58cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:57.557093 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.557034 4861 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30c1072-479f-430a-b0e7-6739582a58cc-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:57.557093 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.557070 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxn92\" (UniqueName: \"kubernetes.io/projected/b30c1072-479f-430a-b0e7-6739582a58cc-kube-api-access-xxn92\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:57.557363 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:57.557102 4861 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b30c1072-479f-430a-b0e7-6739582a58cc-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 16:59:58.171379 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:58.171284 4861 generic.go:356] "Generic (PLEG): container finished" podID="b30c1072-479f-430a-b0e7-6739582a58cc" containerID="cefd959068a104a7a378bd62b56ce1578c96e23e49b094c2de28dfa769445b9f" exitCode=0 Oct 30 16:59:58.172117 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:58.172084 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" event={"ID":"b30c1072-479f-430a-b0e7-6739582a58cc","Type":"ContainerDied","Data":"cefd959068a104a7a378bd62b56ce1578c96e23e49b094c2de28dfa769445b9f"} Oct 30 16:59:58.182218 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:58.182183 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" event={"ID":"b30c1072-479f-430a-b0e7-6739582a58cc","Type":"ContainerDied","Data":"35d19b281807981fc6cca0bc2f3a43e997e97b1595412d0fe36bb63269136add"} Oct 30 16:59:58.182448 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:58.182429 4861 scope.go:122] "RemoveContainer" containerID="cefd959068a104a7a378bd62b56ce1578c96e23e49b094c2de28dfa769445b9f" Oct 30 16:59:58.182702 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:58.172295 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf" Oct 30 16:59:58.215990 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:58.215890 4861 scope.go:122] "RemoveContainer" containerID="cefd959068a104a7a378bd62b56ce1578c96e23e49b094c2de28dfa769445b9f" Oct 30 16:59:58.239838 ip-10-0-131-179 kubenswrapper[4861]: E1030 16:59:58.238148 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefd959068a104a7a378bd62b56ce1578c96e23e49b094c2de28dfa769445b9f\": container with ID starting with cefd959068a104a7a378bd62b56ce1578c96e23e49b094c2de28dfa769445b9f not found: ID does not exist" containerID="cefd959068a104a7a378bd62b56ce1578c96e23e49b094c2de28dfa769445b9f" Oct 30 16:59:58.239838 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:58.238201 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefd959068a104a7a378bd62b56ce1578c96e23e49b094c2de28dfa769445b9f"} err="failed to get container status \"cefd959068a104a7a378bd62b56ce1578c96e23e49b094c2de28dfa769445b9f\": rpc error: code = NotFound desc = could not find container \"cefd959068a104a7a378bd62b56ce1578c96e23e49b094c2de28dfa769445b9f\": container with ID starting with cefd959068a104a7a378bd62b56ce1578c96e23e49b094c2de28dfa769445b9f not found: ID does not exist" Oct 30 16:59:58.259621 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:58.259584 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf"] Oct 30 16:59:58.270237 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:58.270198 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-2035f6/noten-model-2035f6-kserve-6667df986f-hjlbf"] Oct 30 16:59:59.726680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:59.714449 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2"] Oct 30 16:59:59.726680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:59.715534 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b30c1072-479f-430a-b0e7-6739582a58cc" containerName="main" Oct 30 16:59:59.726680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:59.715555 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30c1072-479f-430a-b0e7-6739582a58cc" containerName="main" Oct 30 16:59:59.726680 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:59.715773 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30c1072-479f-430a-b0e7-6739582a58cc" containerName="main" Oct 30 16:59:59.731751 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:59.731138 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" Oct 30 16:59:59.744069 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:59.744020 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2"] Oct 30 16:59:59.747912 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:59.747888 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d5b798\"" Oct 30 16:59:59.763891 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:59.761197 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d5b798-dockercfg-z4vvh\"" Oct 30 16:59:59.796881 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:59.796845 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d2da07-ef07-4524-8627-6e2c5a201e6e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2\" (UID: \"d7d2da07-ef07-4524-8627-6e2c5a201e6e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" Oct 30 16:59:59.797360 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:59.797325 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zc45\" (UniqueName: \"kubernetes.io/projected/d7d2da07-ef07-4524-8627-6e2c5a201e6e-kube-api-access-4zc45\") pod \"payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2\" (UID: \"d7d2da07-ef07-4524-8627-6e2c5a201e6e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" Oct 30 16:59:59.811490 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:59.811462 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30c1072-479f-430a-b0e7-6739582a58cc" path="/var/lib/kubelet/pods/b30c1072-479f-430a-b0e7-6739582a58cc/volumes" Oct 30 16:59:59.898177 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:59.898149 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zc45\" (UniqueName: \"kubernetes.io/projected/d7d2da07-ef07-4524-8627-6e2c5a201e6e-kube-api-access-4zc45\") pod \"payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2\" (UID: \"d7d2da07-ef07-4524-8627-6e2c5a201e6e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" Oct 30 16:59:59.898326 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:59.898217 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d2da07-ef07-4524-8627-6e2c5a201e6e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2\" (UID: \"d7d2da07-ef07-4524-8627-6e2c5a201e6e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" Oct 30 16:59:59.899084 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:59.899059 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d2da07-ef07-4524-8627-6e2c5a201e6e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2\" (UID: \"d7d2da07-ef07-4524-8627-6e2c5a201e6e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" Oct 30 16:59:59.921825 ip-10-0-131-179 kubenswrapper[4861]: I1030 16:59:59.921799 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zc45\" (UniqueName: \"kubernetes.io/projected/d7d2da07-ef07-4524-8627-6e2c5a201e6e-kube-api-access-4zc45\") pod \"payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2\" (UID: \"d7d2da07-ef07-4524-8627-6e2c5a201e6e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" Oct 30 17:00:00.058773 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.058685 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" Oct 30 17:00:00.371193 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.371109 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7"] Oct 30 17:00:00.377165 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.377143 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" Oct 30 17:00:00.386223 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:00:00.386196 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d2da07_ef07_4524_8627_6e2c5a201e6e.slice/crio-3c03831aebb72d2c6223c3058244159205d25d52f8fd3984161f7c6378516dca WatchSource:0}: Error finding container 3c03831aebb72d2c6223c3058244159205d25d52f8fd3984161f7c6378516dca: Status 404 returned error can't find the container with id 3c03831aebb72d2c6223c3058244159205d25d52f8fd3984161f7c6378516dca Oct 30 17:00:00.388285 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.388191 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2"] Oct 30 17:00:00.391895 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.391870 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l"] Oct 30 17:00:00.397739 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.397696 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" Oct 30 17:00:00.402324 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.401839 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z"] Oct 30 17:00:00.411820 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.411794 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:00:00.417850 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.417824 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" Oct 30 17:00:00.425508 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.424818 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7"] Oct 30 17:00:00.429516 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.429493 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l"] Oct 30 17:00:00.430127 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.430108 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z"] Oct 30 17:00:00.514154 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.514121 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef839b9-b1f7-4de6-a436-213aa4e6ab50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7\" (UID: \"bef839b9-b1f7-4de6-a436-213aa4e6ab50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" Oct 30 17:00:00.514498 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.514430 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f55358f-b7a2-4872-a68a-50e75e073aac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l\" (UID: \"8f55358f-b7a2-4872-a68a-50e75e073aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" Oct 30 17:00:00.514656 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.514631 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bssg8\" (UniqueName: \"kubernetes.io/projected/bef839b9-b1f7-4de6-a436-213aa4e6ab50-kube-api-access-bssg8\") pod \"maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7\" (UID: \"bef839b9-b1f7-4de6-a436-213aa4e6ab50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" Oct 30 17:00:00.514792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.514769 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pm6c\" (UniqueName: \"kubernetes.io/projected/8f55358f-b7a2-4872-a68a-50e75e073aac-kube-api-access-4pm6c\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l\" (UID: \"8f55358f-b7a2-4872-a68a-50e75e073aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" Oct 30 17:00:00.616237 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.616206 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f55358f-b7a2-4872-a68a-50e75e073aac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l\" (UID: \"8f55358f-b7a2-4872-a68a-50e75e073aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" Oct 30 17:00:00.616360 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.616309 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bssg8\" (UniqueName: \"kubernetes.io/projected/bef839b9-b1f7-4de6-a436-213aa4e6ab50-kube-api-access-bssg8\") pod \"maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7\" (UID: \"bef839b9-b1f7-4de6-a436-213aa4e6ab50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" Oct 30 17:00:00.616360 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.616343 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pm6c\" (UniqueName: \"kubernetes.io/projected/8f55358f-b7a2-4872-a68a-50e75e073aac-kube-api-access-4pm6c\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l\" (UID: \"8f55358f-b7a2-4872-a68a-50e75e073aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" Oct 30 17:00:00.616442 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.616381 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/182d1ad2-782b-4457-8c18-d5438cd611b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z\" (UID: \"182d1ad2-782b-4457-8c18-d5438cd611b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" Oct 30 17:00:00.616477 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.616452 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef839b9-b1f7-4de6-a436-213aa4e6ab50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7\" (UID: \"bef839b9-b1f7-4de6-a436-213aa4e6ab50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" Oct 30 17:00:00.616531 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.616482 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwgzp\" (UniqueName: \"kubernetes.io/projected/182d1ad2-782b-4457-8c18-d5438cd611b7-kube-api-access-vwgzp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z\" (UID: \"182d1ad2-782b-4457-8c18-d5438cd611b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" Oct 30 17:00:00.616899 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.616877 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f55358f-b7a2-4872-a68a-50e75e073aac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l\" (UID: \"8f55358f-b7a2-4872-a68a-50e75e073aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" Oct 30 17:00:00.617141 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.617110 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef839b9-b1f7-4de6-a436-213aa4e6ab50-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7\" (UID: \"bef839b9-b1f7-4de6-a436-213aa4e6ab50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" Oct 30 17:00:00.640305 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.640255 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pm6c\" (UniqueName: \"kubernetes.io/projected/8f55358f-b7a2-4872-a68a-50e75e073aac-kube-api-access-4pm6c\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l\" (UID: \"8f55358f-b7a2-4872-a68a-50e75e073aac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" Oct 30 17:00:00.640388 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.640325 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bssg8\" (UniqueName: \"kubernetes.io/projected/bef839b9-b1f7-4de6-a436-213aa4e6ab50-kube-api-access-bssg8\") pod \"maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7\" (UID: \"bef839b9-b1f7-4de6-a436-213aa4e6ab50\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" Oct 30 17:00:00.717381 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.717355 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/182d1ad2-782b-4457-8c18-d5438cd611b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z\" (UID: \"182d1ad2-782b-4457-8c18-d5438cd611b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" Oct 30 17:00:00.717495 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.717436 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwgzp\" (UniqueName: \"kubernetes.io/projected/182d1ad2-782b-4457-8c18-d5438cd611b7-kube-api-access-vwgzp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z\" (UID: \"182d1ad2-782b-4457-8c18-d5438cd611b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" Oct 30 17:00:00.717995 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.717971 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/182d1ad2-782b-4457-8c18-d5438cd611b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z\" (UID: \"182d1ad2-782b-4457-8c18-d5438cd611b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" Oct 30 17:00:00.743643 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.743618 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwgzp\" (UniqueName: \"kubernetes.io/projected/182d1ad2-782b-4457-8c18-d5438cd611b7-kube-api-access-vwgzp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z\" (UID: \"182d1ad2-782b-4457-8c18-d5438cd611b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" Oct 30 17:00:00.822509 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.822480 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" Oct 30 17:00:00.847663 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.847630 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" Oct 30 17:00:00.852031 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:00.852011 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" Oct 30 17:00:01.146841 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:01.145900 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7"] Oct 30 17:00:01.181617 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:00:01.181581 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbef839b9_b1f7_4de6_a436_213aa4e6ab50.slice/crio-5f6057cc85cf21e6b383bfe2ff3e3d16354952843fe567524e7b5d84e014dcf4 WatchSource:0}: Error finding container 5f6057cc85cf21e6b383bfe2ff3e3d16354952843fe567524e7b5d84e014dcf4: Status 404 returned error can't find the container with id 5f6057cc85cf21e6b383bfe2ff3e3d16354952843fe567524e7b5d84e014dcf4 Oct 30 17:00:01.200895 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:01.200864 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" event={"ID":"bef839b9-b1f7-4de6-a436-213aa4e6ab50","Type":"ContainerStarted","Data":"5f6057cc85cf21e6b383bfe2ff3e3d16354952843fe567524e7b5d84e014dcf4"} Oct 30 17:00:01.202537 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:01.202509 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" event={"ID":"d7d2da07-ef07-4524-8627-6e2c5a201e6e","Type":"ContainerStarted","Data":"0127b6db41c010658a5d9bba91666853ef3fb5e0410dc056262d20e9c83498c5"} Oct 30 17:00:01.202664 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:01.202554 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" event={"ID":"d7d2da07-ef07-4524-8627-6e2c5a201e6e","Type":"ContainerStarted","Data":"3c03831aebb72d2c6223c3058244159205d25d52f8fd3984161f7c6378516dca"} Oct 30 17:00:01.202750 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:01.202676 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" Oct 30 17:00:01.213411 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:00:01.213387 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182d1ad2_782b_4457_8c18_d5438cd611b7.slice/crio-26bef81e8c91da951a5b558184a6391608f6e89c7aaa01b6889b04141f400125 WatchSource:0}: Error finding container 26bef81e8c91da951a5b558184a6391608f6e89c7aaa01b6889b04141f400125: Status 404 returned error can't find the container with id 26bef81e8c91da951a5b558184a6391608f6e89c7aaa01b6889b04141f400125 Oct 30 17:00:01.213773 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:01.213756 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z"] Oct 30 17:00:01.217906 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:01.217882 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l"] Oct 30 17:00:01.219667 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:00:01.219642 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f55358f_b7a2_4872_a68a_50e75e073aac.slice/crio-a95928d1a1f12f487365f1bc49c17ebc1af1fb1d5008fb43715579096bac8feb WatchSource:0}: Error finding container a95928d1a1f12f487365f1bc49c17ebc1af1fb1d5008fb43715579096bac8feb: Status 404 returned error can't find the container with id a95928d1a1f12f487365f1bc49c17ebc1af1fb1d5008fb43715579096bac8feb Oct 30 17:00:02.215840 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:02.215237 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" event={"ID":"182d1ad2-782b-4457-8c18-d5438cd611b7","Type":"ContainerStarted","Data":"b25fa85aa1c5d079c55555de25a5a3262d560b8a955ca93893cc3743f3852f14"} Oct 30 17:00:02.215840 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:02.215304 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" event={"ID":"182d1ad2-782b-4457-8c18-d5438cd611b7","Type":"ContainerStarted","Data":"26bef81e8c91da951a5b558184a6391608f6e89c7aaa01b6889b04141f400125"} Oct 30 17:00:02.217855 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:02.217829 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" event={"ID":"8f55358f-b7a2-4872-a68a-50e75e073aac","Type":"ContainerStarted","Data":"93eea4a562e3a03ac3f5d71b5c0cc7a03b9f0a373a2ecd47efb28f0eee58f8b3"} Oct 30 17:00:02.217968 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:02.217877 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" event={"ID":"8f55358f-b7a2-4872-a68a-50e75e073aac","Type":"ContainerStarted","Data":"a95928d1a1f12f487365f1bc49c17ebc1af1fb1d5008fb43715579096bac8feb"} Oct 30 17:00:02.218075 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:02.218056 4861 scope.go:122] "RemoveContainer" containerID="93eea4a562e3a03ac3f5d71b5c0cc7a03b9f0a373a2ecd47efb28f0eee58f8b3" Oct 30 17:00:02.219879 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:02.219790 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" event={"ID":"bef839b9-b1f7-4de6-a436-213aa4e6ab50","Type":"ContainerStarted","Data":"333c2a941655547d4028c58849ebd1181f9874f56d21c13957d74d60dc9d4257"} Oct 30 17:00:02.220011 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:02.219995 4861 scope.go:122] "RemoveContainer" containerID="333c2a941655547d4028c58849ebd1181f9874f56d21c13957d74d60dc9d4257" Oct 30 17:00:02.257670 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:02.257614 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" podStartSLOduration=3.257595051 podStartE2EDuration="3.257595051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:01.249887396 +0000 UTC m=+2231.592561437" watchObservedRunningTime="2026-10-30 17:00:02.257595051 +0000 UTC m=+2232.600269089" Oct 30 17:00:02.260801 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:02.260765 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" podStartSLOduration=2.260753443 podStartE2EDuration="2.260753443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.256920986 +0000 UTC m=+2232.599595041" watchObservedRunningTime="2026-10-30 17:00:02.260753443 +0000 UTC m=+2232.603427481" Oct 30 17:00:03.229020 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:03.228982 4861 generic.go:356] "Generic (PLEG): container finished" podID="bef839b9-b1f7-4de6-a436-213aa4e6ab50" containerID="333c2a941655547d4028c58849ebd1181f9874f56d21c13957d74d60dc9d4257" exitCode=60 Oct 30 17:00:03.229513 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:03.229072 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" event={"ID":"bef839b9-b1f7-4de6-a436-213aa4e6ab50","Type":"ContainerDied","Data":"333c2a941655547d4028c58849ebd1181f9874f56d21c13957d74d60dc9d4257"} Oct 30 17:00:03.229513 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:03.229157 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" event={"ID":"bef839b9-b1f7-4de6-a436-213aa4e6ab50","Type":"ContainerStarted","Data":"f0a3f5064babd53d6cb023db090c944c2fa685972fb82a8618e8163e8efcc35a"} Oct 30 17:00:03.230808 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:03.230781 4861 generic.go:356] "Generic (PLEG): container finished" podID="182d1ad2-782b-4457-8c18-d5438cd611b7" containerID="b25fa85aa1c5d079c55555de25a5a3262d560b8a955ca93893cc3743f3852f14" exitCode=60 Oct 30 17:00:03.230920 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:03.230856 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" event={"ID":"182d1ad2-782b-4457-8c18-d5438cd611b7","Type":"ContainerDied","Data":"b25fa85aa1c5d079c55555de25a5a3262d560b8a955ca93893cc3743f3852f14"} Oct 30 17:00:03.231141 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:03.231121 4861 scope.go:122] "RemoveContainer" containerID="b25fa85aa1c5d079c55555de25a5a3262d560b8a955ca93893cc3743f3852f14" Oct 30 17:00:03.232854 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:03.232834 4861 generic.go:356] "Generic (PLEG): container finished" podID="8f55358f-b7a2-4872-a68a-50e75e073aac" containerID="93eea4a562e3a03ac3f5d71b5c0cc7a03b9f0a373a2ecd47efb28f0eee58f8b3" exitCode=60 Oct 30 17:00:03.232934 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:03.232893 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" event={"ID":"8f55358f-b7a2-4872-a68a-50e75e073aac","Type":"ContainerDied","Data":"93eea4a562e3a03ac3f5d71b5c0cc7a03b9f0a373a2ecd47efb28f0eee58f8b3"} Oct 30 17:00:03.233008 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:03.232936 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" event={"ID":"8f55358f-b7a2-4872-a68a-50e75e073aac","Type":"ContainerStarted","Data":"77ae018ecad5a137905ae61dd7d191b74aa8e305510463117202064b5ebc1803"} Oct 30 17:00:03.275448 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:03.275395 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" podStartSLOduration=3.275376988 podStartE2EDuration="3.275376988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:03.27179952 +0000 UTC m=+2233.614473558" watchObservedRunningTime="2026-10-30 17:00:03.275376988 +0000 UTC m=+2233.618051032" Oct 30 17:00:03.315703 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:03.315654 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" podStartSLOduration=3.315639881 podStartE2EDuration="3.315639881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:03.314593671 +0000 UTC m=+2233.657267709" watchObservedRunningTime="2026-10-30 17:00:03.315639881 +0000 UTC m=+2233.658313919" Oct 30 17:00:04.251061 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:04.248712 4861 generic.go:356] "Generic (PLEG): container finished" podID="bef839b9-b1f7-4de6-a436-213aa4e6ab50" containerID="f0a3f5064babd53d6cb023db090c944c2fa685972fb82a8618e8163e8efcc35a" exitCode=60 Oct 30 17:00:04.251061 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:04.248942 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" event={"ID":"bef839b9-b1f7-4de6-a436-213aa4e6ab50","Type":"ContainerDied","Data":"f0a3f5064babd53d6cb023db090c944c2fa685972fb82a8618e8163e8efcc35a"} Oct 30 17:00:04.251061 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:04.249000 4861 scope.go:122] "RemoveContainer" containerID="333c2a941655547d4028c58849ebd1181f9874f56d21c13957d74d60dc9d4257" Oct 30 17:00:04.251061 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:04.249489 4861 scope.go:122] "RemoveContainer" containerID="f0a3f5064babd53d6cb023db090c944c2fa685972fb82a8618e8163e8efcc35a" Oct 30 17:00:04.251061 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:04.249957 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7_odh-ai-gateway-infra(bef839b9-b1f7-4de6-a436-213aa4e6ab50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" podUID="bef839b9-b1f7-4de6-a436-213aa4e6ab50" Oct 30 17:00:04.266032 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:04.263872 4861 generic.go:356] "Generic (PLEG): container finished" podID="182d1ad2-782b-4457-8c18-d5438cd611b7" containerID="9f3bf320d7c8cf4554d577e0ad88ed5c85bb41b88c090d8a75c5d4dc168c1f31" exitCode=60 Oct 30 17:00:04.266032 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:04.263971 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" event={"ID":"182d1ad2-782b-4457-8c18-d5438cd611b7","Type":"ContainerDied","Data":"9f3bf320d7c8cf4554d577e0ad88ed5c85bb41b88c090d8a75c5d4dc168c1f31"} Oct 30 17:00:04.266032 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:04.264458 4861 scope.go:122] "RemoveContainer" containerID="9f3bf320d7c8cf4554d577e0ad88ed5c85bb41b88c090d8a75c5d4dc168c1f31" Oct 30 17:00:04.266032 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:04.264859 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z_odh-ai-gateway-infra(182d1ad2-782b-4457-8c18-d5438cd611b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" podUID="182d1ad2-782b-4457-8c18-d5438cd611b7" Oct 30 17:00:04.279631 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:04.279559 4861 generic.go:356] "Generic (PLEG): container finished" podID="8f55358f-b7a2-4872-a68a-50e75e073aac" containerID="77ae018ecad5a137905ae61dd7d191b74aa8e305510463117202064b5ebc1803" exitCode=60 Oct 30 17:00:04.279942 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:04.279920 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" event={"ID":"8f55358f-b7a2-4872-a68a-50e75e073aac","Type":"ContainerDied","Data":"77ae018ecad5a137905ae61dd7d191b74aa8e305510463117202064b5ebc1803"} Oct 30 17:00:04.280453 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:04.280434 4861 scope.go:122] "RemoveContainer" containerID="77ae018ecad5a137905ae61dd7d191b74aa8e305510463117202064b5ebc1803" Oct 30 17:00:04.281017 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:04.280994 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l_odh-ai-gateway-infra(8f55358f-b7a2-4872-a68a-50e75e073aac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" podUID="8f55358f-b7a2-4872-a68a-50e75e073aac" Oct 30 17:00:04.300927 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:04.300894 4861 scope.go:122] "RemoveContainer" containerID="b25fa85aa1c5d079c55555de25a5a3262d560b8a955ca93893cc3743f3852f14" Oct 30 17:00:04.342532 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:04.339829 4861 scope.go:122] "RemoveContainer" containerID="93eea4a562e3a03ac3f5d71b5c0cc7a03b9f0a373a2ecd47efb28f0eee58f8b3" Oct 30 17:00:05.288697 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:05.288664 4861 scope.go:122] "RemoveContainer" containerID="f0a3f5064babd53d6cb023db090c944c2fa685972fb82a8618e8163e8efcc35a" Oct 30 17:00:05.289166 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:05.289048 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7_odh-ai-gateway-infra(bef839b9-b1f7-4de6-a436-213aa4e6ab50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" podUID="bef839b9-b1f7-4de6-a436-213aa4e6ab50" Oct 30 17:00:05.290361 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:05.290339 4861 scope.go:122] "RemoveContainer" containerID="9f3bf320d7c8cf4554d577e0ad88ed5c85bb41b88c090d8a75c5d4dc168c1f31" Oct 30 17:00:05.290684 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:05.290648 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z_odh-ai-gateway-infra(182d1ad2-782b-4457-8c18-d5438cd611b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" podUID="182d1ad2-782b-4457-8c18-d5438cd611b7" Oct 30 17:00:05.291823 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:05.291801 4861 scope.go:122] "RemoveContainer" containerID="77ae018ecad5a137905ae61dd7d191b74aa8e305510463117202064b5ebc1803" Oct 30 17:00:05.292107 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:05.292086 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l_odh-ai-gateway-infra(8f55358f-b7a2-4872-a68a-50e75e073aac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" podUID="8f55358f-b7a2-4872-a68a-50e75e073aac" Oct 30 17:00:06.188817 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:06.188785 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75"] Oct 30 17:00:07.075860 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.075816 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" podUID="95967951-8ac6-4287-8b0f-9f4a91a6ac0f" containerName="main" containerID="cri-o://efbbb811180ae2b533106299d8b5a5cf1ca3f03afcf8b167a2a2f6d1ee02974e" gracePeriod=45 Oct 30 17:00:07.322221 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.322181 4861 generic.go:356] "Generic (PLEG): container finished" podID="95967951-8ac6-4287-8b0f-9f4a91a6ac0f" containerID="efbbb811180ae2b533106299d8b5a5cf1ca3f03afcf8b167a2a2f6d1ee02974e" exitCode=0 Oct 30 17:00:07.322398 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.322363 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" event={"ID":"95967951-8ac6-4287-8b0f-9f4a91a6ac0f","Type":"ContainerDied","Data":"efbbb811180ae2b533106299d8b5a5cf1ca3f03afcf8b167a2a2f6d1ee02974e"} Oct 30 17:00:07.322398 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.322396 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" event={"ID":"95967951-8ac6-4287-8b0f-9f4a91a6ac0f","Type":"ContainerDied","Data":"4d71a13190905657b1c4930340540ff595c532fe875ffe744a0b49829b1a4480"} Oct 30 17:00:07.322504 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.322410 4861 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d71a13190905657b1c4930340540ff595c532fe875ffe744a0b49829b1a4480" Oct 30 17:00:07.346386 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.346327 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 17:00:07.450099 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.450064 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-kube-api-access-j8rnh\" (UniqueName: \"kubernetes.io/projected/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-kube-api-access-j8rnh\") pod \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " Oct 30 17:00:07.450303 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.450184 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-model-cache\") pod \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " Oct 30 17:00:07.450303 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.450282 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-tls-certs\" (UniqueName: \"kubernetes.io/secret/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-tls-certs\") pod \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " Oct 30 17:00:07.450439 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.450380 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-home\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-home\") pod \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " Oct 30 17:00:07.450516 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.450497 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-dshm\") pod \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " Oct 30 17:00:07.450587 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.450542 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-tmp-dir\") pod \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\" (UID: \"95967951-8ac6-4287-8b0f-9f4a91a6ac0f\") " Oct 30 17:00:07.450967 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.450927 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-model-cache" pod "95967951-8ac6-4287-8b0f-9f4a91a6ac0f" (UID: "95967951-8ac6-4287-8b0f-9f4a91a6ac0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:07.451199 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.451175 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-tmp-dir" pod "95967951-8ac6-4287-8b0f-9f4a91a6ac0f" (UID: "95967951-8ac6-4287-8b0f-9f4a91a6ac0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:07.451199 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.451191 4861 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:07.451758 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.451701 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-home" pod "95967951-8ac6-4287-8b0f-9f4a91a6ac0f" (UID: "95967951-8ac6-4287-8b0f-9f4a91a6ac0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:07.454253 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.454226 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-dshm" pod "95967951-8ac6-4287-8b0f-9f4a91a6ac0f" (UID: "95967951-8ac6-4287-8b0f-9f4a91a6ac0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:07.454378 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.454310 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-kube-api-access-j8rnh" pod "95967951-8ac6-4287-8b0f-9f4a91a6ac0f" (UID: "95967951-8ac6-4287-8b0f-9f4a91a6ac0f"). InnerVolumeSpecName "kube-api-access-j8rnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:07.454897 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.454874 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-tls-certs" pod "95967951-8ac6-4287-8b0f-9f4a91a6ac0f" (UID: "95967951-8ac6-4287-8b0f-9f4a91a6ac0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:07.552775 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.552714 4861 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:07.552881 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.552791 4861 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:07.552881 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.552802 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8rnh\" (UniqueName: \"kubernetes.io/projected/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-kube-api-access-j8rnh\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:07.552881 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.552812 4861 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:07.552881 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:07.552820 4861 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95967951-8ac6-4287-8b0f-9f4a91a6ac0f-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:08.329794 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:08.329761 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn" Oct 30 17:00:08.389007 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:08.388974 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn"] Oct 30 17:00:08.401417 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:08.401389 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cf4621/auth-test-model-cf4621-kserve-85b4f6dddf-n27fn"] Oct 30 17:00:09.797713 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:09.797684 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28"] Oct 30 17:00:09.798347 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:09.798333 4861 cpu_manager.go:414] "containerMap: removing container" podUID="95967951-8ac6-4287-8b0f-9f4a91a6ac0f" containerName="main" Oct 30 17:00:09.798413 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:09.798348 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95967951-8ac6-4287-8b0f-9f4a91a6ac0f" containerName="main" Oct 30 17:00:09.798513 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:09.798503 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="95967951-8ac6-4287-8b0f-9f4a91a6ac0f" containerName="main" Oct 30 17:00:09.803481 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:09.803446 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" Oct 30 17:00:09.815997 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:09.815964 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95967951-8ac6-4287-8b0f-9f4a91a6ac0f" path="/var/lib/kubelet/pods/95967951-8ac6-4287-8b0f-9f4a91a6ac0f/volumes" Oct 30 17:00:09.818797 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:09.818766 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-f94fb5\"" Oct 30 17:00:09.831303 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:09.831276 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-f94fb5-dockercfg-sgrf6\"" Oct 30 17:00:09.836950 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:09.836924 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28"] Oct 30 17:00:09.891140 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:09.891099 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxdxn\" (UniqueName: \"kubernetes.io/projected/b9c6277c-c197-4b2a-945b-9c2553114762-kube-api-access-pxdxn\") pod \"payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28\" (UID: \"b9c6277c-c197-4b2a-945b-9c2553114762\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" Oct 30 17:00:09.891711 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:09.891680 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c6277c-c197-4b2a-945b-9c2553114762-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28\" (UID: \"b9c6277c-c197-4b2a-945b-9c2553114762\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" Oct 30 17:00:09.976830 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:09.976792 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq"] Oct 30 17:00:10.028164 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:10.028102 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c6277c-c197-4b2a-945b-9c2553114762-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28\" (UID: \"b9c6277c-c197-4b2a-945b-9c2553114762\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" Oct 30 17:00:10.028412 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:10.028304 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxdxn\" (UniqueName: \"kubernetes.io/projected/b9c6277c-c197-4b2a-945b-9c2553114762-kube-api-access-pxdxn\") pod \"payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28\" (UID: \"b9c6277c-c197-4b2a-945b-9c2553114762\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" Oct 30 17:00:10.029659 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:10.029623 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c6277c-c197-4b2a-945b-9c2553114762-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28\" (UID: \"b9c6277c-c197-4b2a-945b-9c2553114762\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" Oct 30 17:00:10.036233 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:10.036206 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq"] Oct 30 17:00:10.036373 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:10.036308 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" Oct 30 17:00:10.053328 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:10.053298 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxdxn\" (UniqueName: \"kubernetes.io/projected/b9c6277c-c197-4b2a-945b-9c2553114762-kube-api-access-pxdxn\") pod \"payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28\" (UID: \"b9c6277c-c197-4b2a-945b-9c2553114762\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" Oct 30 17:00:10.121685 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:10.121650 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" Oct 30 17:00:10.129710 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:10.129681 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v6z4\" (UniqueName: \"kubernetes.io/projected/0f099e88-ce47-485b-a7f6-2726c8f128c1-kube-api-access-5v6z4\") pod \"payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq\" (UID: \"0f099e88-ce47-485b-a7f6-2726c8f128c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" Oct 30 17:00:10.129895 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:10.129826 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f099e88-ce47-485b-a7f6-2726c8f128c1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq\" (UID: \"0f099e88-ce47-485b-a7f6-2726c8f128c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" Oct 30 17:00:10.231538 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:10.231486 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v6z4\" (UniqueName: \"kubernetes.io/projected/0f099e88-ce47-485b-a7f6-2726c8f128c1-kube-api-access-5v6z4\") pod \"payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq\" (UID: \"0f099e88-ce47-485b-a7f6-2726c8f128c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" Oct 30 17:00:10.231810 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:10.231602 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f099e88-ce47-485b-a7f6-2726c8f128c1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq\" (UID: \"0f099e88-ce47-485b-a7f6-2726c8f128c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" Oct 30 17:00:10.232706 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:10.232680 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f099e88-ce47-485b-a7f6-2726c8f128c1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq\" (UID: \"0f099e88-ce47-485b-a7f6-2726c8f128c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" Oct 30 17:00:10.257117 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:10.257083 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v6z4\" (UniqueName: \"kubernetes.io/projected/0f099e88-ce47-485b-a7f6-2726c8f128c1-kube-api-access-5v6z4\") pod \"payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq\" (UID: \"0f099e88-ce47-485b-a7f6-2726c8f128c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" Oct 30 17:00:10.372308 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:10.372229 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" Oct 30 17:00:10.430331 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:00:10.430300 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c6277c_c197_4b2a_945b_9c2553114762.slice/crio-2e3b7c1b7837eece24556aded72bcb127e2b6ebdf659810ed99d600706832a66 WatchSource:0}: Error finding container 2e3b7c1b7837eece24556aded72bcb127e2b6ebdf659810ed99d600706832a66: Status 404 returned error can't find the container with id 2e3b7c1b7837eece24556aded72bcb127e2b6ebdf659810ed99d600706832a66 Oct 30 17:00:10.431404 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:10.431377 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28"] Oct 30 17:00:10.689309 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:10.689284 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq"] Oct 30 17:00:10.690068 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:00:10.690039 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f099e88_ce47_485b_a7f6_2726c8f128c1.slice/crio-94883dafb3d29e97ee7ae7c97526641adcecdad9d22d922bd7ddcee8dbb94eec WatchSource:0}: Error finding container 94883dafb3d29e97ee7ae7c97526641adcecdad9d22d922bd7ddcee8dbb94eec: Status 404 returned error can't find the container with id 94883dafb3d29e97ee7ae7c97526641adcecdad9d22d922bd7ddcee8dbb94eec Oct 30 17:00:11.352633 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:11.352594 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" event={"ID":"b9c6277c-c197-4b2a-945b-9c2553114762","Type":"ContainerStarted","Data":"5b29d396eb2dd053e6efc4044eeef4053bef212b1ae834ae26eb3e82d77efb40"} Oct 30 17:00:11.353116 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:11.352656 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" event={"ID":"b9c6277c-c197-4b2a-945b-9c2553114762","Type":"ContainerStarted","Data":"2e3b7c1b7837eece24556aded72bcb127e2b6ebdf659810ed99d600706832a66"} Oct 30 17:00:11.353116 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:11.352688 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" Oct 30 17:00:11.354513 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:11.354486 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" event={"ID":"0f099e88-ce47-485b-a7f6-2726c8f128c1","Type":"ContainerStarted","Data":"5869c05076ce940727db1ad05baacb2608e8c662abb7816d5d6cb4a1d9c1b9f3"} Oct 30 17:00:11.354627 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:11.354606 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" event={"ID":"0f099e88-ce47-485b-a7f6-2726c8f128c1","Type":"ContainerStarted","Data":"94883dafb3d29e97ee7ae7c97526641adcecdad9d22d922bd7ddcee8dbb94eec"} Oct 30 17:00:11.354686 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:11.354641 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" Oct 30 17:00:11.407011 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:11.406952 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" podStartSLOduration=2.406938686 podStartE2EDuration="2.406938686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:11.405745228 +0000 UTC m=+2241.748419260" watchObservedRunningTime="2026-10-30 17:00:11.406938686 +0000 UTC m=+2241.749612752" Oct 30 17:00:11.451768 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:11.451706 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" podStartSLOduration=2.45169469 podStartE2EDuration="2.45169469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:11.450752921 +0000 UTC m=+2241.793426962" watchObservedRunningTime="2026-10-30 17:00:11.45169469 +0000 UTC m=+2241.794368728" Oct 30 17:00:12.224062 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:12.224034 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" Oct 30 17:00:15.800281 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:15.800250 4861 scope.go:122] "RemoveContainer" containerID="9f3bf320d7c8cf4554d577e0ad88ed5c85bb41b88c090d8a75c5d4dc168c1f31" Oct 30 17:00:15.800797 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:15.800356 4861 scope.go:122] "RemoveContainer" containerID="77ae018ecad5a137905ae61dd7d191b74aa8e305510463117202064b5ebc1803" Oct 30 17:00:15.800797 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:15.800472 4861 scope.go:122] "RemoveContainer" containerID="f0a3f5064babd53d6cb023db090c944c2fa685972fb82a8618e8163e8efcc35a" Oct 30 17:00:16.390211 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:16.390127 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" event={"ID":"182d1ad2-782b-4457-8c18-d5438cd611b7","Type":"ContainerStarted","Data":"59a76d772b863445ba8b487e4aca85a264eb71087338b49102e7af7813d5cf51"} Oct 30 17:00:16.392095 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:16.392070 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" event={"ID":"8f55358f-b7a2-4872-a68a-50e75e073aac","Type":"ContainerStarted","Data":"4e34bed32a5e955ffa9a0057b4518b31b58f3de44e51139c9b97bc0dfc36eb16"} Oct 30 17:00:17.053675 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:17.053645 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z"] Oct 30 17:00:17.107801 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:17.107774 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5"] Oct 30 17:00:17.401205 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:17.401129 4861 generic.go:356] "Generic (PLEG): container finished" podID="182d1ad2-782b-4457-8c18-d5438cd611b7" containerID="59a76d772b863445ba8b487e4aca85a264eb71087338b49102e7af7813d5cf51" exitCode=60 Oct 30 17:00:17.401358 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:17.401209 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" event={"ID":"182d1ad2-782b-4457-8c18-d5438cd611b7","Type":"ContainerDied","Data":"59a76d772b863445ba8b487e4aca85a264eb71087338b49102e7af7813d5cf51"} Oct 30 17:00:17.401358 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:17.401269 4861 scope.go:122] "RemoveContainer" containerID="9f3bf320d7c8cf4554d577e0ad88ed5c85bb41b88c090d8a75c5d4dc168c1f31" Oct 30 17:00:17.401763 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:17.401713 4861 scope.go:122] "RemoveContainer" containerID="59a76d772b863445ba8b487e4aca85a264eb71087338b49102e7af7813d5cf51" Oct 30 17:00:17.402146 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:17.402114 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z_odh-ai-gateway-infra(182d1ad2-782b-4457-8c18-d5438cd611b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" podUID="182d1ad2-782b-4457-8c18-d5438cd611b7" Oct 30 17:00:17.403400 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:17.403378 4861 generic.go:356] "Generic (PLEG): container finished" podID="8f55358f-b7a2-4872-a68a-50e75e073aac" containerID="4e34bed32a5e955ffa9a0057b4518b31b58f3de44e51139c9b97bc0dfc36eb16" exitCode=60 Oct 30 17:00:17.403509 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:17.403403 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" event={"ID":"8f55358f-b7a2-4872-a68a-50e75e073aac","Type":"ContainerDied","Data":"4e34bed32a5e955ffa9a0057b4518b31b58f3de44e51139c9b97bc0dfc36eb16"} Oct 30 17:00:17.403804 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:17.403787 4861 scope.go:122] "RemoveContainer" containerID="4e34bed32a5e955ffa9a0057b4518b31b58f3de44e51139c9b97bc0dfc36eb16" Oct 30 17:00:17.404124 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:17.404101 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l_odh-ai-gateway-infra(8f55358f-b7a2-4872-a68a-50e75e073aac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" podUID="8f55358f-b7a2-4872-a68a-50e75e073aac" Oct 30 17:00:17.405431 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:17.405409 4861 generic.go:356] "Generic (PLEG): container finished" podID="bef839b9-b1f7-4de6-a436-213aa4e6ab50" containerID="e4b6717a9adc36cf3967afc35e04927451a23eb50d31e0150fd253c14c785c37" exitCode=60 Oct 30 17:00:17.405526 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:17.405459 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" event={"ID":"bef839b9-b1f7-4de6-a436-213aa4e6ab50","Type":"ContainerDied","Data":"e4b6717a9adc36cf3967afc35e04927451a23eb50d31e0150fd253c14c785c37"} Oct 30 17:00:17.405772 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:17.405708 4861 scope.go:122] "RemoveContainer" containerID="e4b6717a9adc36cf3967afc35e04927451a23eb50d31e0150fd253c14c785c37" Oct 30 17:00:17.405939 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:17.405922 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7_odh-ai-gateway-infra(bef839b9-b1f7-4de6-a436-213aa4e6ab50)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" podUID="bef839b9-b1f7-4de6-a436-213aa4e6ab50" Oct 30 17:00:17.414841 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:17.414821 4861 scope.go:122] "RemoveContainer" containerID="77ae018ecad5a137905ae61dd7d191b74aa8e305510463117202064b5ebc1803" Oct 30 17:00:17.428696 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:17.428678 4861 scope.go:122] "RemoveContainer" containerID="f0a3f5064babd53d6cb023db090c944c2fa685972fb82a8618e8163e8efcc35a" Oct 30 17:00:17.457800 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:17.457683 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z"] Oct 30 17:00:17.501941 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:17.501911 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l"] Oct 30 17:00:18.516854 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.515555 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7"] Oct 30 17:00:18.635267 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.635233 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" Oct 30 17:00:18.675016 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.674926 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f55358f-b7a2-4872-a68a-50e75e073aac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f55358f-b7a2-4872-a68a-50e75e073aac-maas-api-service-ca\") pod \"8f55358f-b7a2-4872-a68a-50e75e073aac\" (UID: \"8f55358f-b7a2-4872-a68a-50e75e073aac\") " Oct 30 17:00:18.675156 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.675018 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f55358f-b7a2-4872-a68a-50e75e073aac-kube-api-access-4pm6c\" (UniqueName: \"kubernetes.io/projected/8f55358f-b7a2-4872-a68a-50e75e073aac-kube-api-access-4pm6c\") pod \"8f55358f-b7a2-4872-a68a-50e75e073aac\" (UID: \"8f55358f-b7a2-4872-a68a-50e75e073aac\") " Oct 30 17:00:18.676876 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.676849 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f55358f-b7a2-4872-a68a-50e75e073aac-maas-api-service-ca" pod "8f55358f-b7a2-4872-a68a-50e75e073aac" (UID: "8f55358f-b7a2-4872-a68a-50e75e073aac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:18.676876 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.676865 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" Oct 30 17:00:18.683907 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.683877 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f55358f-b7a2-4872-a68a-50e75e073aac-kube-api-access-4pm6c" pod "8f55358f-b7a2-4872-a68a-50e75e073aac" (UID: "8f55358f-b7a2-4872-a68a-50e75e073aac"). InnerVolumeSpecName "kube-api-access-4pm6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:18.754219 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.754188 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" Oct 30 17:00:18.777437 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.777410 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bef839b9-b1f7-4de6-a436-213aa4e6ab50-kube-api-access-bssg8\" (UniqueName: \"kubernetes.io/projected/bef839b9-b1f7-4de6-a436-213aa4e6ab50-kube-api-access-bssg8\") pod \"bef839b9-b1f7-4de6-a436-213aa4e6ab50\" (UID: \"bef839b9-b1f7-4de6-a436-213aa4e6ab50\") " Oct 30 17:00:18.777577 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.777508 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bef839b9-b1f7-4de6-a436-213aa4e6ab50-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef839b9-b1f7-4de6-a436-213aa4e6ab50-maas-api-service-ca\") pod \"bef839b9-b1f7-4de6-a436-213aa4e6ab50\" (UID: \"bef839b9-b1f7-4de6-a436-213aa4e6ab50\") " Oct 30 17:00:18.777620 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.777590 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182d1ad2-782b-4457-8c18-d5438cd611b7-kube-api-access-vwgzp\" (UniqueName: \"kubernetes.io/projected/182d1ad2-782b-4457-8c18-d5438cd611b7-kube-api-access-vwgzp\") pod \"182d1ad2-782b-4457-8c18-d5438cd611b7\" (UID: \"182d1ad2-782b-4457-8c18-d5438cd611b7\") " Oct 30 17:00:18.777620 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.777610 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/182d1ad2-782b-4457-8c18-d5438cd611b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/182d1ad2-782b-4457-8c18-d5438cd611b7-maas-api-service-ca\") pod \"182d1ad2-782b-4457-8c18-d5438cd611b7\" (UID: \"182d1ad2-782b-4457-8c18-d5438cd611b7\") " Oct 30 17:00:18.777959 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.777896 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bef839b9-b1f7-4de6-a436-213aa4e6ab50-maas-api-service-ca" pod "bef839b9-b1f7-4de6-a436-213aa4e6ab50" (UID: "bef839b9-b1f7-4de6-a436-213aa4e6ab50"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:18.778102 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.778045 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182d1ad2-782b-4457-8c18-d5438cd611b7-maas-api-service-ca" pod "182d1ad2-782b-4457-8c18-d5438cd611b7" (UID: "182d1ad2-782b-4457-8c18-d5438cd611b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:18.778176 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.778158 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f55358f-b7a2-4872-a68a-50e75e073aac-maas-api-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:18.778242 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.778199 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pm6c\" (UniqueName: \"kubernetes.io/projected/8f55358f-b7a2-4872-a68a-50e75e073aac-kube-api-access-4pm6c\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:18.778242 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.778215 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bef839b9-b1f7-4de6-a436-213aa4e6ab50-maas-api-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:18.778242 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.778228 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/182d1ad2-782b-4457-8c18-d5438cd611b7-maas-api-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:18.779773 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.779750 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef839b9-b1f7-4de6-a436-213aa4e6ab50-kube-api-access-bssg8" pod "bef839b9-b1f7-4de6-a436-213aa4e6ab50" (UID: "bef839b9-b1f7-4de6-a436-213aa4e6ab50"). InnerVolumeSpecName "kube-api-access-bssg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:18.779866 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.779838 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182d1ad2-782b-4457-8c18-d5438cd611b7-kube-api-access-vwgzp" pod "182d1ad2-782b-4457-8c18-d5438cd611b7" (UID: "182d1ad2-782b-4457-8c18-d5438cd611b7"). InnerVolumeSpecName "kube-api-access-vwgzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:18.879101 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.879061 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bssg8\" (UniqueName: \"kubernetes.io/projected/bef839b9-b1f7-4de6-a436-213aa4e6ab50-kube-api-access-bssg8\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:18.879101 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:18.879100 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwgzp\" (UniqueName: \"kubernetes.io/projected/182d1ad2-782b-4457-8c18-d5438cd611b7-kube-api-access-vwgzp\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:19.426953 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.426922 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" Oct 30 17:00:19.427155 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.426967 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7" event={"ID":"bef839b9-b1f7-4de6-a436-213aa4e6ab50","Type":"ContainerDied","Data":"5f6057cc85cf21e6b383bfe2ff3e3d16354952843fe567524e7b5d84e014dcf4"} Oct 30 17:00:19.427155 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.427045 4861 scope.go:122] "RemoveContainer" containerID="e4b6717a9adc36cf3967afc35e04927451a23eb50d31e0150fd253c14c785c37" Oct 30 17:00:19.428756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.428716 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" Oct 30 17:00:19.428756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.428739 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z" event={"ID":"182d1ad2-782b-4457-8c18-d5438cd611b7","Type":"ContainerDied","Data":"26bef81e8c91da951a5b558184a6391608f6e89c7aaa01b6889b04141f400125"} Oct 30 17:00:19.430514 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.430476 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" Oct 30 17:00:19.430636 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.430506 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l" event={"ID":"8f55358f-b7a2-4872-a68a-50e75e073aac","Type":"ContainerDied","Data":"a95928d1a1f12f487365f1bc49c17ebc1af1fb1d5008fb43715579096bac8feb"} Oct 30 17:00:19.439561 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.439539 4861 scope.go:122] "RemoveContainer" containerID="59a76d772b863445ba8b487e4aca85a264eb71087338b49102e7af7813d5cf51" Oct 30 17:00:19.449149 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.449133 4861 scope.go:122] "RemoveContainer" containerID="4e34bed32a5e955ffa9a0057b4518b31b58f3de44e51139c9b97bc0dfc36eb16" Oct 30 17:00:19.484489 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.484464 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l"] Oct 30 17:00:19.495468 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.495442 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-cb6251-29889660-rfg4l"] Oct 30 17:00:19.535633 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.535605 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z"] Oct 30 17:00:19.539555 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.539534 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-8bff6e-29889660-wrd5z"] Oct 30 17:00:19.577634 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.577609 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7"] Oct 30 17:00:19.588122 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.588100 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-18fb02-29889660-qmfp7"] Oct 30 17:00:19.808610 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.808576 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182d1ad2-782b-4457-8c18-d5438cd611b7" path="/var/lib/kubelet/pods/182d1ad2-782b-4457-8c18-d5438cd611b7/volumes" Oct 30 17:00:19.809029 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.809013 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f55358f-b7a2-4872-a68a-50e75e073aac" path="/var/lib/kubelet/pods/8f55358f-b7a2-4872-a68a-50e75e073aac/volumes" Oct 30 17:00:19.809433 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:19.809419 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bef839b9-b1f7-4de6-a436-213aa4e6ab50" path="/var/lib/kubelet/pods/bef839b9-b1f7-4de6-a436-213aa4e6ab50/volumes" Oct 30 17:00:20.320514 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:20.320481 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:00:21.211425 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.211361 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" podUID="eeb7ae54-914d-4f58-a85f-e360f8082832" containerName="main" containerID="cri-o://e97fd15d3f4f7c53736b583e83ec111b79ce000269d3f0456d4a894a61ab8863" gracePeriod=45 Oct 30 17:00:21.455329 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.455296 4861 generic.go:356] "Generic (PLEG): container finished" podID="eeb7ae54-914d-4f58-a85f-e360f8082832" containerID="e97fd15d3f4f7c53736b583e83ec111b79ce000269d3f0456d4a894a61ab8863" exitCode=0 Oct 30 17:00:21.455475 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.455369 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" event={"ID":"eeb7ae54-914d-4f58-a85f-e360f8082832","Type":"ContainerDied","Data":"e97fd15d3f4f7c53736b583e83ec111b79ce000269d3f0456d4a894a61ab8863"} Oct 30 17:00:21.512270 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.512245 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 17:00:21.665801 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.665764 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-dshm\") pod \"eeb7ae54-914d-4f58-a85f-e360f8082832\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " Oct 30 17:00:21.665971 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.665826 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-tmp-dir\") pod \"eeb7ae54-914d-4f58-a85f-e360f8082832\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " Oct 30 17:00:21.665971 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.665871 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-model-cache\") pod \"eeb7ae54-914d-4f58-a85f-e360f8082832\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " Oct 30 17:00:21.665971 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.665910 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeb7ae54-914d-4f58-a85f-e360f8082832-kube-api-access-cbd9m\" (UniqueName: \"kubernetes.io/projected/eeb7ae54-914d-4f58-a85f-e360f8082832-kube-api-access-cbd9m\") pod \"eeb7ae54-914d-4f58-a85f-e360f8082832\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " Oct 30 17:00:21.666129 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.666113 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb7ae54-914d-4f58-a85f-e360f8082832-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb7ae54-914d-4f58-a85f-e360f8082832-tls-certs\") pod \"eeb7ae54-914d-4f58-a85f-e360f8082832\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " Oct 30 17:00:21.666185 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.666139 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-model-cache" pod "eeb7ae54-914d-4f58-a85f-e360f8082832" (UID: "eeb7ae54-914d-4f58-a85f-e360f8082832"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:21.666185 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.666153 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-home\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-home\") pod \"eeb7ae54-914d-4f58-a85f-e360f8082832\" (UID: \"eeb7ae54-914d-4f58-a85f-e360f8082832\") " Oct 30 17:00:21.666296 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.666151 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-tmp-dir" pod "eeb7ae54-914d-4f58-a85f-e360f8082832" (UID: "eeb7ae54-914d-4f58-a85f-e360f8082832"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:21.666371 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.666347 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-home" pod "eeb7ae54-914d-4f58-a85f-e360f8082832" (UID: "eeb7ae54-914d-4f58-a85f-e360f8082832"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:21.666673 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.666641 4861 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.666673 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.666664 4861 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.666673 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.666678 4861 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.668269 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.668243 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-dshm" pod "eeb7ae54-914d-4f58-a85f-e360f8082832" (UID: "eeb7ae54-914d-4f58-a85f-e360f8082832"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:21.668470 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.668449 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb7ae54-914d-4f58-a85f-e360f8082832-kube-api-access-cbd9m" pod "eeb7ae54-914d-4f58-a85f-e360f8082832" (UID: "eeb7ae54-914d-4f58-a85f-e360f8082832"). InnerVolumeSpecName "kube-api-access-cbd9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:21.668545 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.668453 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb7ae54-914d-4f58-a85f-e360f8082832-tls-certs" pod "eeb7ae54-914d-4f58-a85f-e360f8082832" (UID: "eeb7ae54-914d-4f58-a85f-e360f8082832"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:21.767535 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.767469 4861 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb7ae54-914d-4f58-a85f-e360f8082832-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.767535 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.767494 4861 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb7ae54-914d-4f58-a85f-e360f8082832-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.767535 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:21.767507 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbd9m\" (UniqueName: \"kubernetes.io/projected/eeb7ae54-914d-4f58-a85f-e360f8082832-kube-api-access-cbd9m\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.054630 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.054523 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" podUID="06557e2c-fcca-4155-8322-5b666975f8be" containerName="payload-processing" containerID="cri-o://3f462b06ffdb8da5af4bdcb5611b31f544778d90cd51485f6721589f267c8e1b" gracePeriod=25 Oct 30 17:00:22.108333 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.108304 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" podUID="c84db179-8068-4c19-be7e-228a37b89a2c" containerName="payload-pre-processing" containerID="cri-o://87adfd3480e86814e07a781bc667ed9adc8d036f84c1a65ae4c39d2e75f374b0" gracePeriod=25 Oct 30 17:00:22.365328 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.365296 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" Oct 30 17:00:22.367431 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.367406 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" Oct 30 17:00:22.393887 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.393861 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" Oct 30 17:00:22.445773 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.444014 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" Oct 30 17:00:22.467645 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.467612 4861 generic.go:356] "Generic (PLEG): container finished" podID="c84db179-8068-4c19-be7e-228a37b89a2c" containerID="87adfd3480e86814e07a781bc667ed9adc8d036f84c1a65ae4c39d2e75f374b0" exitCode=0 Oct 30 17:00:22.469240 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.469206 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" event={"ID":"c84db179-8068-4c19-be7e-228a37b89a2c","Type":"ContainerDied","Data":"87adfd3480e86814e07a781bc667ed9adc8d036f84c1a65ae4c39d2e75f374b0"} Oct 30 17:00:22.469366 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.469275 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" event={"ID":"c84db179-8068-4c19-be7e-228a37b89a2c","Type":"ContainerDied","Data":"acb9103008a9cf1019c96306c261fb273c8a4649dc12fc2ab61eb0dcce8833a5"} Oct 30 17:00:22.469366 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.469304 4861 scope.go:122] "RemoveContainer" containerID="87adfd3480e86814e07a781bc667ed9adc8d036f84c1a65ae4c39d2e75f374b0" Oct 30 17:00:22.469505 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.469436 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5" Oct 30 17:00:22.472685 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.472654 4861 generic.go:356] "Generic (PLEG): container finished" podID="06557e2c-fcca-4155-8322-5b666975f8be" containerID="3f462b06ffdb8da5af4bdcb5611b31f544778d90cd51485f6721589f267c8e1b" exitCode=0 Oct 30 17:00:22.472866 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.472845 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" event={"ID":"06557e2c-fcca-4155-8322-5b666975f8be","Type":"ContainerDied","Data":"3f462b06ffdb8da5af4bdcb5611b31f544778d90cd51485f6721589f267c8e1b"} Oct 30 17:00:22.472938 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.472890 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" event={"ID":"06557e2c-fcca-4155-8322-5b666975f8be","Type":"ContainerDied","Data":"54fa6b3d8344c4a25b15e21b3d9af37f2da4e751c70ae3e5cd5f380ead7cfed2"} Oct 30 17:00:22.473003 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.472971 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z" Oct 30 17:00:22.474838 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.474812 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06557e2c-fcca-4155-8322-5b666975f8be-kube-api-access-rxwjz\" (UniqueName: \"kubernetes.io/projected/06557e2c-fcca-4155-8322-5b666975f8be-kube-api-access-rxwjz\") pod \"06557e2c-fcca-4155-8322-5b666975f8be\" (UID: \"06557e2c-fcca-4155-8322-5b666975f8be\") " Oct 30 17:00:22.474990 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.474969 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06557e2c-fcca-4155-8322-5b666975f8be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06557e2c-fcca-4155-8322-5b666975f8be-plugins-config-volume\") pod \"06557e2c-fcca-4155-8322-5b666975f8be\" (UID: \"06557e2c-fcca-4155-8322-5b666975f8be\") " Oct 30 17:00:22.478351 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.478325 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06557e2c-fcca-4155-8322-5b666975f8be-plugins-config-volume" pod "06557e2c-fcca-4155-8322-5b666975f8be" (UID: "06557e2c-fcca-4155-8322-5b666975f8be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:22.478596 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.478577 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" Oct 30 17:00:22.478682 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.478645 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75" event={"ID":"eeb7ae54-914d-4f58-a85f-e360f8082832","Type":"ContainerDied","Data":"c5278a95d314b225f2fd7aca1b789984ac2283fb6efbe76244828278c3d3681a"} Oct 30 17:00:22.481881 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.481812 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06557e2c-fcca-4155-8322-5b666975f8be-kube-api-access-rxwjz" pod "06557e2c-fcca-4155-8322-5b666975f8be" (UID: "06557e2c-fcca-4155-8322-5b666975f8be"). InnerVolumeSpecName "kube-api-access-rxwjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:22.485045 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.485024 4861 scope.go:122] "RemoveContainer" containerID="87adfd3480e86814e07a781bc667ed9adc8d036f84c1a65ae4c39d2e75f374b0" Oct 30 17:00:22.485385 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:22.485353 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87adfd3480e86814e07a781bc667ed9adc8d036f84c1a65ae4c39d2e75f374b0\": container with ID starting with 87adfd3480e86814e07a781bc667ed9adc8d036f84c1a65ae4c39d2e75f374b0 not found: ID does not exist" containerID="87adfd3480e86814e07a781bc667ed9adc8d036f84c1a65ae4c39d2e75f374b0" Oct 30 17:00:22.485479 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.485391 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87adfd3480e86814e07a781bc667ed9adc8d036f84c1a65ae4c39d2e75f374b0"} err="failed to get container status \"87adfd3480e86814e07a781bc667ed9adc8d036f84c1a65ae4c39d2e75f374b0\": rpc error: code = NotFound desc = could not find container \"87adfd3480e86814e07a781bc667ed9adc8d036f84c1a65ae4c39d2e75f374b0\": container with ID starting with 87adfd3480e86814e07a781bc667ed9adc8d036f84c1a65ae4c39d2e75f374b0 not found: ID does not exist" Oct 30 17:00:22.485479 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.485407 4861 scope.go:122] "RemoveContainer" containerID="3f462b06ffdb8da5af4bdcb5611b31f544778d90cd51485f6721589f267c8e1b" Oct 30 17:00:22.498654 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.497914 4861 scope.go:122] "RemoveContainer" containerID="3f462b06ffdb8da5af4bdcb5611b31f544778d90cd51485f6721589f267c8e1b" Oct 30 17:00:22.502023 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:22.502001 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f462b06ffdb8da5af4bdcb5611b31f544778d90cd51485f6721589f267c8e1b\": container with ID starting with 3f462b06ffdb8da5af4bdcb5611b31f544778d90cd51485f6721589f267c8e1b not found: ID does not exist" containerID="3f462b06ffdb8da5af4bdcb5611b31f544778d90cd51485f6721589f267c8e1b" Oct 30 17:00:22.502135 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.502028 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f462b06ffdb8da5af4bdcb5611b31f544778d90cd51485f6721589f267c8e1b"} err="failed to get container status \"3f462b06ffdb8da5af4bdcb5611b31f544778d90cd51485f6721589f267c8e1b\": rpc error: code = NotFound desc = could not find container \"3f462b06ffdb8da5af4bdcb5611b31f544778d90cd51485f6721589f267c8e1b\": container with ID starting with 3f462b06ffdb8da5af4bdcb5611b31f544778d90cd51485f6721589f267c8e1b not found: ID does not exist" Oct 30 17:00:22.502135 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.502048 4861 scope.go:122] "RemoveContainer" containerID="e97fd15d3f4f7c53736b583e83ec111b79ce000269d3f0456d4a894a61ab8863" Oct 30 17:00:22.576967 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.576919 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c84db179-8068-4c19-be7e-228a37b89a2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c84db179-8068-4c19-be7e-228a37b89a2c-plugins-config-volume\") pod \"c84db179-8068-4c19-be7e-228a37b89a2c\" (UID: \"c84db179-8068-4c19-be7e-228a37b89a2c\") " Oct 30 17:00:22.577138 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.576975 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c84db179-8068-4c19-be7e-228a37b89a2c-kube-api-access-pn8zd\" (UniqueName: \"kubernetes.io/projected/c84db179-8068-4c19-be7e-228a37b89a2c-kube-api-access-pn8zd\") pod \"c84db179-8068-4c19-be7e-228a37b89a2c\" (UID: \"c84db179-8068-4c19-be7e-228a37b89a2c\") " Oct 30 17:00:22.577332 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.577306 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84db179-8068-4c19-be7e-228a37b89a2c-plugins-config-volume" pod "c84db179-8068-4c19-be7e-228a37b89a2c" (UID: "c84db179-8068-4c19-be7e-228a37b89a2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:22.577669 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.577645 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06557e2c-fcca-4155-8322-5b666975f8be-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.577803 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.577674 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxwjz\" (UniqueName: \"kubernetes.io/projected/06557e2c-fcca-4155-8322-5b666975f8be-kube-api-access-rxwjz\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.577803 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.577692 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c84db179-8068-4c19-be7e-228a37b89a2c-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.579606 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.579584 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84db179-8068-4c19-be7e-228a37b89a2c-kube-api-access-pn8zd" pod "c84db179-8068-4c19-be7e-228a37b89a2c" (UID: "c84db179-8068-4c19-be7e-228a37b89a2c"). InnerVolumeSpecName "kube-api-access-pn8zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:22.622074 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.612390 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75"] Oct 30 17:00:22.636928 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.631554 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-18fb02/explicit-model-18fb02-kserve-84758fffbf-bjl75"] Oct 30 17:00:22.678874 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.678842 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn8zd\" (UniqueName: \"kubernetes.io/projected/c84db179-8068-4c19-be7e-228a37b89a2c-kube-api-access-pn8zd\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.823988 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.823957 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5"] Oct 30 17:00:22.829111 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.829080 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-18fb02-787f747f5f-kdls5"] Oct 30 17:00:22.863482 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.863459 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z"] Oct 30 17:00:22.867967 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:22.867946 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-18fb02-6f7486ffd4-wqt7z"] Oct 30 17:00:23.818354 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:23.818311 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06557e2c-fcca-4155-8322-5b666975f8be" path="/var/lib/kubelet/pods/06557e2c-fcca-4155-8322-5b666975f8be/volumes" Oct 30 17:00:23.818955 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:23.818930 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c84db179-8068-4c19-be7e-228a37b89a2c" path="/var/lib/kubelet/pods/c84db179-8068-4c19-be7e-228a37b89a2c/volumes" Oct 30 17:00:23.819580 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:23.819519 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeb7ae54-914d-4f58-a85f-e360f8082832" path="/var/lib/kubelet/pods/eeb7ae54-914d-4f58-a85f-e360f8082832/volumes" Oct 30 17:00:24.233541 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.233508 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk"] Oct 30 17:00:24.234143 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234128 4861 cpu_manager.go:414] "containerMap: removing container" podUID="182d1ad2-782b-4457-8c18-d5438cd611b7" containerName="cleanup" Oct 30 17:00:24.234206 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234144 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182d1ad2-782b-4457-8c18-d5438cd611b7" containerName="cleanup" Oct 30 17:00:24.234206 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234157 4861 cpu_manager.go:414] "containerMap: removing container" podUID="182d1ad2-782b-4457-8c18-d5438cd611b7" containerName="cleanup" Oct 30 17:00:24.234206 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234162 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182d1ad2-782b-4457-8c18-d5438cd611b7" containerName="cleanup" Oct 30 17:00:24.234206 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234172 4861 cpu_manager.go:414] "containerMap: removing container" podUID="c84db179-8068-4c19-be7e-228a37b89a2c" containerName="payload-pre-processing" Oct 30 17:00:24.234206 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234177 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c84db179-8068-4c19-be7e-228a37b89a2c" containerName="payload-pre-processing" Oct 30 17:00:24.234206 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234184 4861 cpu_manager.go:414] "containerMap: removing container" podUID="bef839b9-b1f7-4de6-a436-213aa4e6ab50" containerName="cleanup" Oct 30 17:00:24.234206 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234188 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef839b9-b1f7-4de6-a436-213aa4e6ab50" containerName="cleanup" Oct 30 17:00:24.234206 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234197 4861 cpu_manager.go:414] "containerMap: removing container" podUID="06557e2c-fcca-4155-8322-5b666975f8be" containerName="payload-processing" Oct 30 17:00:24.234206 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234202 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06557e2c-fcca-4155-8322-5b666975f8be" containerName="payload-processing" Oct 30 17:00:24.234206 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234209 4861 cpu_manager.go:414] "containerMap: removing container" podUID="8f55358f-b7a2-4872-a68a-50e75e073aac" containerName="cleanup" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234213 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f55358f-b7a2-4872-a68a-50e75e073aac" containerName="cleanup" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234222 4861 cpu_manager.go:414] "containerMap: removing container" podUID="8f55358f-b7a2-4872-a68a-50e75e073aac" containerName="cleanup" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234226 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f55358f-b7a2-4872-a68a-50e75e073aac" containerName="cleanup" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234243 4861 cpu_manager.go:414] "containerMap: removing container" podUID="8f55358f-b7a2-4872-a68a-50e75e073aac" containerName="cleanup" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234249 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f55358f-b7a2-4872-a68a-50e75e073aac" containerName="cleanup" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234257 4861 cpu_manager.go:414] "containerMap: removing container" podUID="eeb7ae54-914d-4f58-a85f-e360f8082832" containerName="main" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234262 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb7ae54-914d-4f58-a85f-e360f8082832" containerName="main" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234279 4861 cpu_manager.go:414] "containerMap: removing container" podUID="bef839b9-b1f7-4de6-a436-213aa4e6ab50" containerName="cleanup" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234284 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef839b9-b1f7-4de6-a436-213aa4e6ab50" containerName="cleanup" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234387 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef839b9-b1f7-4de6-a436-213aa4e6ab50" containerName="cleanup" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234396 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef839b9-b1f7-4de6-a436-213aa4e6ab50" containerName="cleanup" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234404 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="182d1ad2-782b-4457-8c18-d5438cd611b7" containerName="cleanup" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234410 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="182d1ad2-782b-4457-8c18-d5438cd611b7" containerName="cleanup" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234416 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="c84db179-8068-4c19-be7e-228a37b89a2c" containerName="payload-pre-processing" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234422 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="bef839b9-b1f7-4de6-a436-213aa4e6ab50" containerName="cleanup" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234427 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="06557e2c-fcca-4155-8322-5b666975f8be" containerName="payload-processing" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234433 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f55358f-b7a2-4872-a68a-50e75e073aac" containerName="cleanup" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234438 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f55358f-b7a2-4872-a68a-50e75e073aac" containerName="cleanup" Oct 30 17:00:24.234518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.234446 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb7ae54-914d-4f58-a85f-e360f8082832" containerName="main" Oct 30 17:00:24.239636 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.239616 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:00:24.259009 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.258970 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk"] Oct 30 17:00:24.268239 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.267616 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d5b798-serving-cert\"" Oct 30 17:00:24.399279 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.399251 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6724k\" (UniqueName: \"kubernetes.io/projected/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-kube-api-access-6724k\") pod \"maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk\" (UID: \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:00:24.399441 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.399298 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-metrics-certs\") pod \"maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk\" (UID: \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:00:24.399504 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.399462 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk\" (UID: \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:00:24.399504 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.399498 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk\" (UID: \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:00:24.509848 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.509753 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-metrics-certs\") pod \"maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk\" (UID: \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:00:24.510389 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.510359 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk\" (UID: \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:00:24.510568 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.510550 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk\" (UID: \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:00:24.511021 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.510988 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6724k\" (UniqueName: \"kubernetes.io/projected/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-kube-api-access-6724k\") pod \"maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk\" (UID: \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:00:24.516739 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.512477 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk\" (UID: \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:00:24.526526 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.526499 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk\" (UID: \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:00:24.536364 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.536312 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-metrics-certs\") pod \"maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk\" (UID: \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:00:24.541472 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.539789 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6724k\" (UniqueName: \"kubernetes.io/projected/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-kube-api-access-6724k\") pod \"maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk\" (UID: \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:00:24.553535 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.553504 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:00:24.873450 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.873421 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk"] Oct 30 17:00:24.874505 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:00:24.874480 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ca4500_8d2b_4e41_8ae2_13f452ded9c3.slice/crio-e1a9363e673d774e2d4902781b5f9c9b21fa78601fb3c70816973365cfdf8bb9 WatchSource:0}: Error finding container e1a9363e673d774e2d4902781b5f9c9b21fa78601fb3c70816973365cfdf8bb9: Status 404 returned error can't find the container with id e1a9363e673d774e2d4902781b5f9c9b21fa78601fb3c70816973365cfdf8bb9 Oct 30 17:00:24.876000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:24.875980 4861 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:00:25.572204 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:25.567299 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" event={"ID":"29ca4500-8d2b-4e41-8ae2-13f452ded9c3","Type":"ContainerStarted","Data":"9339b658b2039763aaa35c7cc7eba7c1e37c45aa853bf55bf20db54bedee743b"} Oct 30 17:00:25.572204 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:25.567367 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" event={"ID":"29ca4500-8d2b-4e41-8ae2-13f452ded9c3","Type":"ContainerStarted","Data":"e1a9363e673d774e2d4902781b5f9c9b21fa78601fb3c70816973365cfdf8bb9"} Oct 30 17:00:25.572204 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:25.568517 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:00:25.635256 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:25.635202 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" podStartSLOduration=1.215709473 podStartE2EDuration="1.635187006s" totalImagesPullingTime="419.477533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:25.632273979 +0000 UTC m=+2255.974948019" watchObservedRunningTime="2026-10-30 17:00:25.635187006 +0000 UTC m=+2255.977861041" Oct 30 17:00:28.672917 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:28.669536 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj"] Oct 30 17:00:28.672917 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:28.669827 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" podUID="a4187598-e0e1-4787-87ad-d7eb1f0af775" containerName="istio-proxy" containerID="cri-o://fda01eadaf6dd2e13b17245450a5a39e4114799d97eac30fc6ba5d0777ee3669" gracePeriod=30 Oct 30 17:00:28.708264 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:28.708227 4861 patch_prober.go:29] interesting pod/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:00:28.708383 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:28.708279 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" podUID="a4187598-e0e1-4787-87ad-d7eb1f0af775" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:00:30.007844 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:30.007812 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7428-3beb34"] Oct 30 17:00:30.008457 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:30.008441 4861 cpu_manager.go:414] "containerMap: removing container" podUID="182d1ad2-782b-4457-8c18-d5438cd611b7" containerName="cleanup" Oct 30 17:00:30.008521 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:30.008458 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182d1ad2-782b-4457-8c18-d5438cd611b7" containerName="cleanup" Oct 30 17:00:30.008521 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:30.008470 4861 cpu_manager.go:414] "containerMap: removing container" podUID="bef839b9-b1f7-4de6-a436-213aa4e6ab50" containerName="cleanup" Oct 30 17:00:30.008521 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:30.008475 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bef839b9-b1f7-4de6-a436-213aa4e6ab50" containerName="cleanup" Oct 30 17:00:30.008623 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:30.008597 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="182d1ad2-782b-4457-8c18-d5438cd611b7" containerName="cleanup" Oct 30 17:00:30.008623 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:30.008606 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f55358f-b7a2-4872-a68a-50e75e073aac" containerName="cleanup" Oct 30 17:00:30.012480 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:30.012459 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7428-3beb34" Oct 30 17:00:30.030301 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:30.030278 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7428-3beb34"] Oct 30 17:00:30.038987 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:30.038964 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qn5rq\"" Oct 30 17:00:30.120166 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:30.120138 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvpvm\" (UniqueName: \"kubernetes.io/projected/a5e95c8c-d522-46e8-a387-b87c21df3a97-kube-api-access-nvpvm\") pod \"test-curl-7428-3beb34\" (UID: \"a5e95c8c-d522-46e8-a387-b87c21df3a97\") " pod="opendatahub/test-curl-7428-3beb34" Oct 30 17:00:30.220862 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:30.220837 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvpvm\" (UniqueName: \"kubernetes.io/projected/a5e95c8c-d522-46e8-a387-b87c21df3a97-kube-api-access-nvpvm\") pod \"test-curl-7428-3beb34\" (UID: \"a5e95c8c-d522-46e8-a387-b87c21df3a97\") " pod="opendatahub/test-curl-7428-3beb34" Oct 30 17:00:30.245009 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:30.244987 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvpvm\" (UniqueName: \"kubernetes.io/projected/a5e95c8c-d522-46e8-a387-b87c21df3a97-kube-api-access-nvpvm\") pod \"test-curl-7428-3beb34\" (UID: \"a5e95c8c-d522-46e8-a387-b87c21df3a97\") " pod="opendatahub/test-curl-7428-3beb34" Oct 30 17:00:30.326277 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:30.326187 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7428-3beb34" Oct 30 17:00:30.646204 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:30.646172 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7428-3beb34"] Oct 30 17:00:31.579942 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:31.579915 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:00:31.614411 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:31.614379 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7428-3beb34" event={"ID":"a5e95c8c-d522-46e8-a387-b87c21df3a97","Type":"ContainerStarted","Data":"510ec975b1de129491271db54bfd27cc798945d09006ae128b367313f59fb944"} Oct 30 17:00:31.614589 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:31.614425 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7428-3beb34" event={"ID":"a5e95c8c-d522-46e8-a387-b87c21df3a97","Type":"ContainerStarted","Data":"f9092ca131a25508a942d654476fd59d6ba1370b5baaf4285df2b396f1f844ac"} Oct 30 17:00:31.666512 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:31.666453 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7428-3beb34" podStartSLOduration=2.666439666 podStartE2EDuration="2.666439666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:31.662434194 +0000 UTC m=+2262.005108232" watchObservedRunningTime="2026-10-30 17:00:31.666439666 +0000 UTC m=+2262.009113704" Oct 30 17:00:32.564874 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:32.564834 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7428-3beb34"] Oct 30 17:00:32.570617 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:32.570588 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7428-3beb34"] Oct 30 17:00:33.630194 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:33.630135 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7428-3beb34" podUID="a5e95c8c-d522-46e8-a387-b87c21df3a97" containerName="test-curl-7428-3beb34" containerID="cri-o://510ec975b1de129491271db54bfd27cc798945d09006ae128b367313f59fb944" gracePeriod=2 Oct 30 17:00:33.645878 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:33.645836 4861 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7428-3beb34\" is forbidden: User \"system:node:ip-10-0-131-179.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-179.ec2.internal' and this object" podUID="a5e95c8c-d522-46e8-a387-b87c21df3a97" pod="opendatahub/test-curl-7428-3beb34" Oct 30 17:00:33.871638 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:33.871603 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw"] Oct 30 17:00:33.872666 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:33.872638 4861 cpu_manager.go:414] "containerMap: removing container" podUID="a5e95c8c-d522-46e8-a387-b87c21df3a97" containerName="test-curl-7428-3beb34" Oct 30 17:00:33.872666 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:33.872663 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e95c8c-d522-46e8-a387-b87c21df3a97" containerName="test-curl-7428-3beb34" Oct 30 17:00:33.872953 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:33.872924 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e95c8c-d522-46e8-a387-b87c21df3a97" containerName="test-curl-7428-3beb34" Oct 30 17:00:33.877306 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:33.877284 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" Oct 30 17:00:33.901948 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:33.901580 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw"] Oct 30 17:00:33.906447 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:33.904097 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2589b5\"" Oct 30 17:00:33.906575 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:33.904120 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-2589b5-dockercfg-lfc2l\"" Oct 30 17:00:33.995559 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:33.995515 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2spf\" (UniqueName: \"kubernetes.io/projected/3939645c-9b16-4a51-91a2-e5f043ddcf9f-kube-api-access-p2spf\") pod \"payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw\" (UID: \"3939645c-9b16-4a51-91a2-e5f043ddcf9f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" Oct 30 17:00:33.995919 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:33.995898 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3939645c-9b16-4a51-91a2-e5f043ddcf9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw\" (UID: \"3939645c-9b16-4a51-91a2-e5f043ddcf9f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" Oct 30 17:00:34.010829 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.010786 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk"] Oct 30 17:00:34.032426 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.032381 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" Oct 30 17:00:34.100541 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.098193 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bndzq\" (UniqueName: \"kubernetes.io/projected/dda01ef9-7338-4724-ba7f-b9fe73cdfd4a-kube-api-access-bndzq\") pod \"payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk\" (UID: \"dda01ef9-7338-4724-ba7f-b9fe73cdfd4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" Oct 30 17:00:34.100541 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.098234 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda01ef9-7338-4724-ba7f-b9fe73cdfd4a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk\" (UID: \"dda01ef9-7338-4724-ba7f-b9fe73cdfd4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" Oct 30 17:00:34.100541 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.098303 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2spf\" (UniqueName: \"kubernetes.io/projected/3939645c-9b16-4a51-91a2-e5f043ddcf9f-kube-api-access-p2spf\") pod \"payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw\" (UID: \"3939645c-9b16-4a51-91a2-e5f043ddcf9f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" Oct 30 17:00:34.100541 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.098344 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3939645c-9b16-4a51-91a2-e5f043ddcf9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw\" (UID: \"3939645c-9b16-4a51-91a2-e5f043ddcf9f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" Oct 30 17:00:34.100541 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.100233 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3939645c-9b16-4a51-91a2-e5f043ddcf9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw\" (UID: \"3939645c-9b16-4a51-91a2-e5f043ddcf9f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" Oct 30 17:00:34.106377 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.103882 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk"] Oct 30 17:00:34.125269 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.125245 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 17:00:34.126945 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.126925 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2spf\" (UniqueName: \"kubernetes.io/projected/3939645c-9b16-4a51-91a2-e5f043ddcf9f-kube-api-access-p2spf\") pod \"payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw\" (UID: \"3939645c-9b16-4a51-91a2-e5f043ddcf9f\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" Oct 30 17:00:34.199337 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.199259 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-data\") pod \"a4187598-e0e1-4787-87ad-d7eb1f0af775\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " Oct 30 17:00:34.199337 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.199308 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-podinfo\") pod \"a4187598-e0e1-4787-87ad-d7eb1f0af775\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " Oct 30 17:00:34.199567 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.199350 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4187598-e0e1-4787-87ad-d7eb1f0af775-kube-api-access-xdbhg\" (UniqueName: \"kubernetes.io/projected/a4187598-e0e1-4787-87ad-d7eb1f0af775-kube-api-access-xdbhg\") pod \"a4187598-e0e1-4787-87ad-d7eb1f0af775\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " Oct 30 17:00:34.199567 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.199384 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-credential-socket\") pod \"a4187598-e0e1-4787-87ad-d7eb1f0af775\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " Oct 30 17:00:34.199567 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.199425 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-token\" (UniqueName: \"kubernetes.io/projected/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-token\") pod \"a4187598-e0e1-4787-87ad-d7eb1f0af775\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " Oct 30 17:00:34.199567 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.199455 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-workload-socket\") pod \"a4187598-e0e1-4787-87ad-d7eb1f0af775\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " Oct 30 17:00:34.199567 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.199512 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-envoy\") pod \"a4187598-e0e1-4787-87ad-d7eb1f0af775\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " Oct 30 17:00:34.199567 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.199517 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-data" pod "a4187598-e0e1-4787-87ad-d7eb1f0af775" (UID: "a4187598-e0e1-4787-87ad-d7eb1f0af775"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:34.199567 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.199550 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-workload-certs\") pod \"a4187598-e0e1-4787-87ad-d7eb1f0af775\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " Oct 30 17:00:34.199948 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.199635 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4187598-e0e1-4787-87ad-d7eb1f0af775-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4187598-e0e1-4787-87ad-d7eb1f0af775-istiod-ca-cert\") pod \"a4187598-e0e1-4787-87ad-d7eb1f0af775\" (UID: \"a4187598-e0e1-4787-87ad-d7eb1f0af775\") " Oct 30 17:00:34.199948 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.199736 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-credential-socket" pod "a4187598-e0e1-4787-87ad-d7eb1f0af775" (UID: "a4187598-e0e1-4787-87ad-d7eb1f0af775"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:34.199948 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.199768 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-workload-socket" pod "a4187598-e0e1-4787-87ad-d7eb1f0af775" (UID: "a4187598-e0e1-4787-87ad-d7eb1f0af775"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:34.200183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.199970 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-workload-certs" pod "a4187598-e0e1-4787-87ad-d7eb1f0af775" (UID: "a4187598-e0e1-4787-87ad-d7eb1f0af775"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:34.200183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.200067 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bndzq\" (UniqueName: \"kubernetes.io/projected/dda01ef9-7338-4724-ba7f-b9fe73cdfd4a-kube-api-access-bndzq\") pod \"payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk\" (UID: \"dda01ef9-7338-4724-ba7f-b9fe73cdfd4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" Oct 30 17:00:34.200183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.200121 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda01ef9-7338-4724-ba7f-b9fe73cdfd4a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk\" (UID: \"dda01ef9-7338-4724-ba7f-b9fe73cdfd4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" Oct 30 17:00:34.200364 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.200186 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4187598-e0e1-4787-87ad-d7eb1f0af775-istiod-ca-cert" pod "a4187598-e0e1-4787-87ad-d7eb1f0af775" (UID: "a4187598-e0e1-4787-87ad-d7eb1f0af775"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:34.200364 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.200353 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:34.200468 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.200373 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:34.200468 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.200386 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:34.200468 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.200395 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4187598-e0e1-4787-87ad-d7eb1f0af775-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:34.200468 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.200405 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:34.201437 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.201395 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda01ef9-7338-4724-ba7f-b9fe73cdfd4a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk\" (UID: \"dda01ef9-7338-4724-ba7f-b9fe73cdfd4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" Oct 30 17:00:34.202854 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.202426 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" Oct 30 17:00:34.202854 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.202786 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-token" pod "a4187598-e0e1-4787-87ad-d7eb1f0af775" (UID: "a4187598-e0e1-4787-87ad-d7eb1f0af775"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:34.203336 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.203294 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-envoy" pod "a4187598-e0e1-4787-87ad-d7eb1f0af775" (UID: "a4187598-e0e1-4787-87ad-d7eb1f0af775"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:34.217132 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.217093 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4187598-e0e1-4787-87ad-d7eb1f0af775-kube-api-access-xdbhg" pod "a4187598-e0e1-4787-87ad-d7eb1f0af775" (UID: "a4187598-e0e1-4787-87ad-d7eb1f0af775"). InnerVolumeSpecName "kube-api-access-xdbhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:34.217240 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.217176 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-podinfo" pod "a4187598-e0e1-4787-87ad-d7eb1f0af775" (UID: "a4187598-e0e1-4787-87ad-d7eb1f0af775"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:00:34.224382 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.224350 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bndzq\" (UniqueName: \"kubernetes.io/projected/dda01ef9-7338-4724-ba7f-b9fe73cdfd4a-kube-api-access-bndzq\") pod \"payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk\" (UID: \"dda01ef9-7338-4724-ba7f-b9fe73cdfd4a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" Oct 30 17:00:34.301278 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.301249 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:34.301278 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.301278 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdbhg\" (UniqueName: \"kubernetes.io/projected/a4187598-e0e1-4787-87ad-d7eb1f0af775-kube-api-access-xdbhg\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:34.301455 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.301287 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:34.301455 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.301295 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4187598-e0e1-4787-87ad-d7eb1f0af775-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:34.419153 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.419125 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" Oct 30 17:00:34.539988 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:00:34.539951 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3939645c_9b16_4a51_91a2_e5f043ddcf9f.slice/crio-ac36a353829ae2796f8eb6ca54ee7ea1db9893b8b8ef4246ee190c14ecc978da WatchSource:0}: Error finding container ac36a353829ae2796f8eb6ca54ee7ea1db9893b8b8ef4246ee190c14ecc978da: Status 404 returned error can't find the container with id ac36a353829ae2796f8eb6ca54ee7ea1db9893b8b8ef4246ee190c14ecc978da Oct 30 17:00:34.540709 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.540679 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw"] Oct 30 17:00:34.639187 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.639151 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" event={"ID":"3939645c-9b16-4a51-91a2-e5f043ddcf9f","Type":"ContainerStarted","Data":"f85fd9bd017ef5f63d9bdc2fb0768e800fe921d8ba1305f4f8ec7086cb32a7d1"} Oct 30 17:00:34.639583 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.639212 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" event={"ID":"3939645c-9b16-4a51-91a2-e5f043ddcf9f","Type":"ContainerStarted","Data":"ac36a353829ae2796f8eb6ca54ee7ea1db9893b8b8ef4246ee190c14ecc978da"} Oct 30 17:00:34.639583 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.639233 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" Oct 30 17:00:34.640979 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.640953 4861 generic.go:356] "Generic (PLEG): container finished" podID="a4187598-e0e1-4787-87ad-d7eb1f0af775" containerID="fda01eadaf6dd2e13b17245450a5a39e4114799d97eac30fc6ba5d0777ee3669" exitCode=0 Oct 30 17:00:34.641120 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.641009 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" Oct 30 17:00:34.641120 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.641042 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" event={"ID":"a4187598-e0e1-4787-87ad-d7eb1f0af775","Type":"ContainerDied","Data":"fda01eadaf6dd2e13b17245450a5a39e4114799d97eac30fc6ba5d0777ee3669"} Oct 30 17:00:34.641234 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.641123 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj" event={"ID":"a4187598-e0e1-4787-87ad-d7eb1f0af775","Type":"ContainerDied","Data":"81d10c23c3f1d981dc31d768984c720de8bc0ca03f05e98bf22182367f088596"} Oct 30 17:00:34.641234 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.641156 4861 scope.go:122] "RemoveContainer" containerID="fda01eadaf6dd2e13b17245450a5a39e4114799d97eac30fc6ba5d0777ee3669" Oct 30 17:00:34.651996 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.651977 4861 scope.go:122] "RemoveContainer" containerID="feb4db1f6bc55c5eec3c96521dd7f30045d129b682620b41a0a79152a823dfe4" Oct 30 17:00:34.661982 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.661963 4861 scope.go:122] "RemoveContainer" containerID="fda01eadaf6dd2e13b17245450a5a39e4114799d97eac30fc6ba5d0777ee3669" Oct 30 17:00:34.662218 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:34.662199 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda01eadaf6dd2e13b17245450a5a39e4114799d97eac30fc6ba5d0777ee3669\": container with ID starting with fda01eadaf6dd2e13b17245450a5a39e4114799d97eac30fc6ba5d0777ee3669 not found: ID does not exist" containerID="fda01eadaf6dd2e13b17245450a5a39e4114799d97eac30fc6ba5d0777ee3669" Oct 30 17:00:34.662278 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.662229 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda01eadaf6dd2e13b17245450a5a39e4114799d97eac30fc6ba5d0777ee3669"} err="failed to get container status \"fda01eadaf6dd2e13b17245450a5a39e4114799d97eac30fc6ba5d0777ee3669\": rpc error: code = NotFound desc = could not find container \"fda01eadaf6dd2e13b17245450a5a39e4114799d97eac30fc6ba5d0777ee3669\": container with ID starting with fda01eadaf6dd2e13b17245450a5a39e4114799d97eac30fc6ba5d0777ee3669 not found: ID does not exist" Oct 30 17:00:34.662278 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.662249 4861 scope.go:122] "RemoveContainer" containerID="feb4db1f6bc55c5eec3c96521dd7f30045d129b682620b41a0a79152a823dfe4" Oct 30 17:00:34.662483 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:34.662465 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb4db1f6bc55c5eec3c96521dd7f30045d129b682620b41a0a79152a823dfe4\": container with ID starting with feb4db1f6bc55c5eec3c96521dd7f30045d129b682620b41a0a79152a823dfe4 not found: ID does not exist" containerID="feb4db1f6bc55c5eec3c96521dd7f30045d129b682620b41a0a79152a823dfe4" Oct 30 17:00:34.662534 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.662488 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb4db1f6bc55c5eec3c96521dd7f30045d129b682620b41a0a79152a823dfe4"} err="failed to get container status \"feb4db1f6bc55c5eec3c96521dd7f30045d129b682620b41a0a79152a823dfe4\": rpc error: code = NotFound desc = could not find container \"feb4db1f6bc55c5eec3c96521dd7f30045d129b682620b41a0a79152a823dfe4\": container with ID starting with feb4db1f6bc55c5eec3c96521dd7f30045d129b682620b41a0a79152a823dfe4 not found: ID does not exist" Oct 30 17:00:34.682638 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.682584 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" podStartSLOduration=1.6825720739999999 podStartE2EDuration="1.682572074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:34.681377343 +0000 UTC m=+2265.024051381" watchObservedRunningTime="2026-10-30 17:00:34.682572074 +0000 UTC m=+2265.025246137" Oct 30 17:00:34.720290 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.720263 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj"] Oct 30 17:00:34.725021 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.724954 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-18fb02-openshift-default-547576b45-rwrnj"] Oct 30 17:00:34.729349 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:34.729289 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk"] Oct 30 17:00:34.732870 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:00:34.732846 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda01ef9_7338_4724_ba7f_b9fe73cdfd4a.slice/crio-f4a6aa5df3a8bfd81636a71f8416cd60d0fe60317c0ae6fe408c0fb7ef9d7ec3 WatchSource:0}: Error finding container f4a6aa5df3a8bfd81636a71f8416cd60d0fe60317c0ae6fe408c0fb7ef9d7ec3: Status 404 returned error can't find the container with id f4a6aa5df3a8bfd81636a71f8416cd60d0fe60317c0ae6fe408c0fb7ef9d7ec3 Oct 30 17:00:35.661052 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:35.661005 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" event={"ID":"dda01ef9-7338-4724-ba7f-b9fe73cdfd4a","Type":"ContainerStarted","Data":"4dbb8d84f95251643ff0e435d0ef8e39770586c93bfff2820a9fc2c7332b3700"} Oct 30 17:00:35.661501 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:35.661098 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" event={"ID":"dda01ef9-7338-4724-ba7f-b9fe73cdfd4a","Type":"ContainerStarted","Data":"f4a6aa5df3a8bfd81636a71f8416cd60d0fe60317c0ae6fe408c0fb7ef9d7ec3"} Oct 30 17:00:35.661501 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:35.661170 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" Oct 30 17:00:35.811976 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:35.811941 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4187598-e0e1-4787-87ad-d7eb1f0af775" path="/var/lib/kubelet/pods/a4187598-e0e1-4787-87ad-d7eb1f0af775/volumes" Oct 30 17:00:35.859172 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:35.859146 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7428-3beb34" Oct 30 17:00:35.920692 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:35.920618 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e95c8c-d522-46e8-a387-b87c21df3a97-kube-api-access-nvpvm\" (UniqueName: \"kubernetes.io/projected/a5e95c8c-d522-46e8-a387-b87c21df3a97-kube-api-access-nvpvm\") pod \"a5e95c8c-d522-46e8-a387-b87c21df3a97\" (UID: \"a5e95c8c-d522-46e8-a387-b87c21df3a97\") " Oct 30 17:00:35.922930 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:35.922901 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e95c8c-d522-46e8-a387-b87c21df3a97-kube-api-access-nvpvm" pod "a5e95c8c-d522-46e8-a387-b87c21df3a97" (UID: "a5e95c8c-d522-46e8-a387-b87c21df3a97"). InnerVolumeSpecName "kube-api-access-nvpvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:36.022420 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.022386 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvpvm\" (UniqueName: \"kubernetes.io/projected/a5e95c8c-d522-46e8-a387-b87c21df3a97-kube-api-access-nvpvm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:36.225193 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.225136 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" podStartSLOduration=3.225121072 podStartE2EDuration="3.225121072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:35.711424886 +0000 UTC m=+2266.054098928" watchObservedRunningTime="2026-10-30 17:00:36.225121072 +0000 UTC m=+2266.567795110" Oct 30 17:00:36.228533 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.228507 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f"] Oct 30 17:00:36.229188 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.229174 4861 cpu_manager.go:414] "containerMap: removing container" podUID="a4187598-e0e1-4787-87ad-d7eb1f0af775" containerName="istio-proxy" Oct 30 17:00:36.229233 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.229189 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4187598-e0e1-4787-87ad-d7eb1f0af775" containerName="istio-proxy" Oct 30 17:00:36.229344 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.229333 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4187598-e0e1-4787-87ad-d7eb1f0af775" containerName="istio-proxy" Oct 30 17:00:36.229381 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.229350 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4187598-e0e1-4787-87ad-d7eb1f0af775" containerName="istio-proxy" Oct 30 17:00:36.232934 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.232918 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:00:36.262141 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.262114 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-f94fb5-serving-cert\"" Oct 30 17:00:36.266085 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.266059 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f"] Oct 30 17:00:36.326986 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.326948 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f988c028-5967-4436-a8cd-30af1e920b40-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f\" (UID: \"f988c028-5967-4436-a8cd-30af1e920b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:00:36.327141 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.327031 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5872\" (UniqueName: \"kubernetes.io/projected/f988c028-5967-4436-a8cd-30af1e920b40-kube-api-access-l5872\") pod \"maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f\" (UID: \"f988c028-5967-4436-a8cd-30af1e920b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:00:36.327211 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.327130 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f988c028-5967-4436-a8cd-30af1e920b40-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f\" (UID: \"f988c028-5967-4436-a8cd-30af1e920b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:00:36.327397 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.327374 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f988c028-5967-4436-a8cd-30af1e920b40-metrics-certs\") pod \"maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f\" (UID: \"f988c028-5967-4436-a8cd-30af1e920b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:00:36.428907 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.428872 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f988c028-5967-4436-a8cd-30af1e920b40-metrics-certs\") pod \"maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f\" (UID: \"f988c028-5967-4436-a8cd-30af1e920b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:00:36.429044 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.428946 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f988c028-5967-4436-a8cd-30af1e920b40-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f\" (UID: \"f988c028-5967-4436-a8cd-30af1e920b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:00:36.429044 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.429001 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5872\" (UniqueName: \"kubernetes.io/projected/f988c028-5967-4436-a8cd-30af1e920b40-kube-api-access-l5872\") pod \"maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f\" (UID: \"f988c028-5967-4436-a8cd-30af1e920b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:00:36.429152 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.429106 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f988c028-5967-4436-a8cd-30af1e920b40-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f\" (UID: \"f988c028-5967-4436-a8cd-30af1e920b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:00:36.429753 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.429706 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f988c028-5967-4436-a8cd-30af1e920b40-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f\" (UID: \"f988c028-5967-4436-a8cd-30af1e920b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:00:36.431509 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.431488 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f988c028-5967-4436-a8cd-30af1e920b40-metrics-certs\") pod \"maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f\" (UID: \"f988c028-5967-4436-a8cd-30af1e920b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:00:36.431598 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.431540 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f988c028-5967-4436-a8cd-30af1e920b40-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f\" (UID: \"f988c028-5967-4436-a8cd-30af1e920b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:00:36.453003 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.452977 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5872\" (UniqueName: \"kubernetes.io/projected/f988c028-5967-4436-a8cd-30af1e920b40-kube-api-access-l5872\") pod \"maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f\" (UID: \"f988c028-5967-4436-a8cd-30af1e920b40\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:00:36.547767 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.547655 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:00:36.672960 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.672923 4861 generic.go:356] "Generic (PLEG): container finished" podID="a5e95c8c-d522-46e8-a387-b87c21df3a97" containerID="510ec975b1de129491271db54bfd27cc798945d09006ae128b367313f59fb944" exitCode=137 Oct 30 17:00:36.673421 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.673001 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7428-3beb34" Oct 30 17:00:36.673421 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.673018 4861 scope.go:122] "RemoveContainer" containerID="510ec975b1de129491271db54bfd27cc798945d09006ae128b367313f59fb944" Oct 30 17:00:36.686861 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.686687 4861 scope.go:122] "RemoveContainer" containerID="510ec975b1de129491271db54bfd27cc798945d09006ae128b367313f59fb944" Oct 30 17:00:36.687186 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:36.687086 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510ec975b1de129491271db54bfd27cc798945d09006ae128b367313f59fb944\": container with ID starting with 510ec975b1de129491271db54bfd27cc798945d09006ae128b367313f59fb944 not found: ID does not exist" containerID="510ec975b1de129491271db54bfd27cc798945d09006ae128b367313f59fb944" Oct 30 17:00:36.687186 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.687118 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510ec975b1de129491271db54bfd27cc798945d09006ae128b367313f59fb944"} err="failed to get container status \"510ec975b1de129491271db54bfd27cc798945d09006ae128b367313f59fb944\": rpc error: code = NotFound desc = could not find container \"510ec975b1de129491271db54bfd27cc798945d09006ae128b367313f59fb944\": container with ID starting with 510ec975b1de129491271db54bfd27cc798945d09006ae128b367313f59fb944 not found: ID does not exist" Oct 30 17:00:36.699793 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:36.699765 4861 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7428-3beb34\" is forbidden: User \"system:node:ip-10-0-131-179.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-179.ec2.internal' and this object" podUID="a5e95c8c-d522-46e8-a387-b87c21df3a97" pod="opendatahub/test-curl-7428-3beb34" Oct 30 17:00:36.854931 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:36.854909 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f"] Oct 30 17:00:36.857253 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:00:36.857222 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf988c028_5967_4436_a8cd_30af1e920b40.slice/crio-7cbab083596685678fd9b54c7c20aa8e3760c71b9eb1006cebb0d7a9c61ca02c WatchSource:0}: Error finding container 7cbab083596685678fd9b54c7c20aa8e3760c71b9eb1006cebb0d7a9c61ca02c: Status 404 returned error can't find the container with id 7cbab083596685678fd9b54c7c20aa8e3760c71b9eb1006cebb0d7a9c61ca02c Oct 30 17:00:37.684712 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:37.684677 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" event={"ID":"f988c028-5967-4436-a8cd-30af1e920b40","Type":"ContainerStarted","Data":"a0b5fa829a0d52f40f28a9ea8b8802943a54208491b8d8aed3e6012a9a239b47"} Oct 30 17:00:37.685210 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:37.684755 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" event={"ID":"f988c028-5967-4436-a8cd-30af1e920b40","Type":"ContainerStarted","Data":"7cbab083596685678fd9b54c7c20aa8e3760c71b9eb1006cebb0d7a9c61ca02c"} Oct 30 17:00:37.685210 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:37.684786 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:00:37.729566 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:37.729512 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" podStartSLOduration=1.4274908179999999 podStartE2EDuration="1.729498583s" totalImagesPullingTime="302.007765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:37.728054696 +0000 UTC m=+2268.070728747" watchObservedRunningTime="2026-10-30 17:00:37.729498583 +0000 UTC m=+2268.072172620" Oct 30 17:00:37.742420 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:37.742389 4861 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7428-3beb34\" is forbidden: User \"system:node:ip-10-0-131-179.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-179.ec2.internal' and this object" podUID="a5e95c8c-d522-46e8-a387-b87c21df3a97" pod="opendatahub/test-curl-7428-3beb34" Oct 30 17:00:37.808912 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:37.808889 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e95c8c-d522-46e8-a387-b87c21df3a97" path="/var/lib/kubelet/pods/a5e95c8c-d522-46e8-a387-b87c21df3a97/volumes" Oct 30 17:00:39.439944 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:39.439907 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s"] Oct 30 17:00:39.440693 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:39.440674 4861 cpu_manager.go:414] "containerMap: removing container" podUID="a4187598-e0e1-4787-87ad-d7eb1f0af775" containerName="istio-proxy" Oct 30 17:00:39.440779 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:39.440695 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4187598-e0e1-4787-87ad-d7eb1f0af775" containerName="istio-proxy" Oct 30 17:00:39.445598 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:39.445579 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" Oct 30 17:00:39.464549 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:39.464524 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s"] Oct 30 17:00:39.473006 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:39.472982 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-3106d6\"" Oct 30 17:00:39.586743 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:39.586680 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be98f9c8-1b20-40fa-84f5-74f69b2271c5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s\" (UID: \"be98f9c8-1b20-40fa-84f5-74f69b2271c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" Oct 30 17:00:39.595286 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:39.595234 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwg56\" (UniqueName: \"kubernetes.io/projected/be98f9c8-1b20-40fa-84f5-74f69b2271c5-kube-api-access-vwg56\") pod \"payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s\" (UID: \"be98f9c8-1b20-40fa-84f5-74f69b2271c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" Oct 30 17:00:39.698094 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:39.698012 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwg56\" (UniqueName: \"kubernetes.io/projected/be98f9c8-1b20-40fa-84f5-74f69b2271c5-kube-api-access-vwg56\") pod \"payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s\" (UID: \"be98f9c8-1b20-40fa-84f5-74f69b2271c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" Oct 30 17:00:39.698242 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:39.698128 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be98f9c8-1b20-40fa-84f5-74f69b2271c5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s\" (UID: \"be98f9c8-1b20-40fa-84f5-74f69b2271c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" Oct 30 17:00:39.698841 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:39.698816 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be98f9c8-1b20-40fa-84f5-74f69b2271c5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s\" (UID: \"be98f9c8-1b20-40fa-84f5-74f69b2271c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" Oct 30 17:00:39.724263 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:39.724232 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwg56\" (UniqueName: \"kubernetes.io/projected/be98f9c8-1b20-40fa-84f5-74f69b2271c5-kube-api-access-vwg56\") pod \"payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s\" (UID: \"be98f9c8-1b20-40fa-84f5-74f69b2271c5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" Oct 30 17:00:39.760124 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:39.760091 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" Oct 30 17:00:40.075139 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:40.075110 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s"] Oct 30 17:00:40.076476 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:00:40.076447 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe98f9c8_1b20_40fa_84f5_74f69b2271c5.slice/crio-d70c1d98bda9e308cc104bd9b8dbb1cbe1761829310306318eaa10b3acb5da1c WatchSource:0}: Error finding container d70c1d98bda9e308cc104bd9b8dbb1cbe1761829310306318eaa10b3acb5da1c: Status 404 returned error can't find the container with id d70c1d98bda9e308cc104bd9b8dbb1cbe1761829310306318eaa10b3acb5da1c Oct 30 17:00:40.709266 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:40.709229 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" event={"ID":"be98f9c8-1b20-40fa-84f5-74f69b2271c5","Type":"ContainerStarted","Data":"27483c960fb63cd3bdfe2b9d46c952f8102077b667a73c5b77cbcdbe0cfb9aea"} Oct 30 17:00:40.709690 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:40.709287 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" event={"ID":"be98f9c8-1b20-40fa-84f5-74f69b2271c5","Type":"ContainerStarted","Data":"d70c1d98bda9e308cc104bd9b8dbb1cbe1761829310306318eaa10b3acb5da1c"} Oct 30 17:00:40.709690 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:40.709314 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" Oct 30 17:00:40.754270 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:40.754217 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" podStartSLOduration=1.754202582 podStartE2EDuration="1.754202582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:40.750984655 +0000 UTC m=+2271.093658695" watchObservedRunningTime="2026-10-30 17:00:40.754202582 +0000 UTC m=+2271.096876620" Oct 30 17:00:43.696477 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:43.696447 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:00:45.665794 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:45.665757 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" Oct 30 17:00:46.676522 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:46.676497 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" Oct 30 17:00:47.252435 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:47.252403 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc"] Oct 30 17:00:47.252705 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:47.252594 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" podUID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" containerName="istio-proxy" containerID="cri-o://20ac9166683516cc981d646eb5e5859f896a853a7563fba4ceb2c6c133ccd392" gracePeriod=30 Oct 30 17:00:50.291038 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:50.290982 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:00:51.721327 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:51.721291 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" Oct 30 17:00:52.541219 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.541187 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 17:00:52.662320 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.662281 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-token\" (UniqueName: \"kubernetes.io/projected/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-token\") pod \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " Oct 30 17:00:52.662320 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.662324 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-envoy\") pod \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " Oct 30 17:00:52.662572 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.662347 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-podinfo\") pod \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " Oct 30 17:00:52.662572 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.662392 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-data\") pod \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " Oct 30 17:00:52.662572 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.662458 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istiod-ca-cert\") pod \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " Oct 30 17:00:52.662572 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.662485 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-kube-api-access-7l82g\" (UniqueName: \"kubernetes.io/projected/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-kube-api-access-7l82g\") pod \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " Oct 30 17:00:52.662572 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.662516 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-workload-certs\") pod \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " Oct 30 17:00:52.662929 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.662768 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-data" pod "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" (UID: "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:52.662929 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.662834 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-workload-socket\") pod \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " Oct 30 17:00:52.663045 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.662962 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-workload-certs" pod "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" (UID: "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:52.663045 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.663009 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-credential-socket\") pod \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\" (UID: \"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2\") " Oct 30 17:00:52.663546 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.663089 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istiod-ca-cert" pod "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" (UID: "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:52.663546 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.663409 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-workload-socket" pod "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" (UID: "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:52.663546 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.663534 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-credential-socket" pod "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" (UID: "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:52.664042 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.664017 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:52.664129 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.664046 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:52.664129 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.664062 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:52.664129 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.664077 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:52.664129 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.664091 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:52.665072 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.665039 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-envoy" pod "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" (UID: "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:52.665072 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.665042 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-podinfo" pod "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" (UID: "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:00:52.665255 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.665140 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-token" pod "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" (UID: "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:52.665375 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.665360 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-kube-api-access-7l82g" pod "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" (UID: "dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2"). InnerVolumeSpecName "kube-api-access-7l82g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:52.765424 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.765390 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:52.765424 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.765423 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:52.765878 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.765436 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:52.765878 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.765448 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l82g\" (UniqueName: \"kubernetes.io/projected/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2-kube-api-access-7l82g\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:00:52.799670 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.799591 4861 generic.go:356] "Generic (PLEG): container finished" podID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" containerID="20ac9166683516cc981d646eb5e5859f896a853a7563fba4ceb2c6c133ccd392" exitCode=0 Oct 30 17:00:52.799670 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.799634 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" Oct 30 17:00:52.799919 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.799665 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" event={"ID":"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2","Type":"ContainerDied","Data":"20ac9166683516cc981d646eb5e5859f896a853a7563fba4ceb2c6c133ccd392"} Oct 30 17:00:52.799919 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.799743 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc" event={"ID":"dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2","Type":"ContainerDied","Data":"6ae63ea72ded0a2ff42e408bb8e9393c0fed950fefde54dc7903c62b7844d6b5"} Oct 30 17:00:52.799919 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.799769 4861 scope.go:122] "RemoveContainer" containerID="20ac9166683516cc981d646eb5e5859f896a853a7563fba4ceb2c6c133ccd392" Oct 30 17:00:52.811340 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.811322 4861 scope.go:122] "RemoveContainer" containerID="20ac9166683516cc981d646eb5e5859f896a853a7563fba4ceb2c6c133ccd392" Oct 30 17:00:52.811636 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:00:52.811619 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20ac9166683516cc981d646eb5e5859f896a853a7563fba4ceb2c6c133ccd392\": container with ID starting with 20ac9166683516cc981d646eb5e5859f896a853a7563fba4ceb2c6c133ccd392 not found: ID does not exist" containerID="20ac9166683516cc981d646eb5e5859f896a853a7563fba4ceb2c6c133ccd392" Oct 30 17:00:52.811715 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.811642 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20ac9166683516cc981d646eb5e5859f896a853a7563fba4ceb2c6c133ccd392"} err="failed to get container status \"20ac9166683516cc981d646eb5e5859f896a853a7563fba4ceb2c6c133ccd392\": rpc error: code = NotFound desc = could not find container \"20ac9166683516cc981d646eb5e5859f896a853a7563fba4ceb2c6c133ccd392\": container with ID starting with 20ac9166683516cc981d646eb5e5859f896a853a7563fba4ceb2c6c133ccd392 not found: ID does not exist" Oct 30 17:00:52.855691 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.855667 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc"] Oct 30 17:00:52.860830 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:52.860805 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-xcccc"] Oct 30 17:00:53.809224 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:53.809195 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" path="/var/lib/kubelet/pods/dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2/volumes" Oct 30 17:00:55.477710 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:55.477662 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq"] Oct 30 17:00:56.030291 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.030259 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89"] Oct 30 17:00:56.031220 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.031188 4861 cpu_manager.go:414] "containerMap: removing container" podUID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" containerName="istio-proxy" Oct 30 17:00:56.031220 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.031215 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" containerName="istio-proxy" Oct 30 17:00:56.031418 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.031402 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcdcc450-d8cb-4d93-b4ae-0ed5dff857f2" containerName="istio-proxy" Oct 30 17:00:56.036570 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.036550 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.052061 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.052034 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-06e7f6-openshift-default-dockercfg-pf8rn\"" Oct 30 17:00:56.068551 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.068524 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89"] Oct 30 17:00:56.109303 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.109278 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-credential-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.109468 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.109315 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-workload-certs\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.109553 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.109492 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48fd7\" (UniqueName: \"kubernetes.io/projected/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-kube-api-access-48fd7\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.109553 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.109540 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-token\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.109635 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.109614 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-podinfo\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.109693 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.109648 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istiod-ca-cert\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.109827 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.109803 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-data\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.109929 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.109901 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-envoy\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.110041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.109932 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-workload-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.211453 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.211425 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-data\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.211587 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.211487 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-envoy\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.211587 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.211509 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-workload-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.211587 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.211557 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-credential-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.211587 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.211581 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-workload-certs\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.211851 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.211645 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48fd7\" (UniqueName: \"kubernetes.io/projected/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-kube-api-access-48fd7\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.211851 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.211665 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-token\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.211851 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.211718 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-podinfo\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.211851 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.211791 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istiod-ca-cert\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.212090 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.211914 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-data\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.212090 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.211968 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-workload-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.212090 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.212074 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-credential-socket\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.212358 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.212336 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-workload-certs\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.212456 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.212435 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istiod-ca-cert\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.214169 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.214147 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-envoy\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.214624 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.214594 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-podinfo\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.235501 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.235479 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-token\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.235666 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.235646 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48fd7\" (UniqueName: \"kubernetes.io/projected/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-kube-api-access-48fd7\") pod \"e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.350898 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.350814 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:56.697547 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.697517 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89"] Oct 30 17:00:56.707606 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.707568 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:00:56.707761 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.707658 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:00:56.707761 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.707700 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:00:56.835808 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.835777 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" event={"ID":"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf","Type":"ContainerStarted","Data":"a542a3e78a6d628f01397c66eacde0362422c9bbc2b0b2ab6a84d563204696c2"} Oct 30 17:00:56.836007 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.835993 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" event={"ID":"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf","Type":"ContainerStarted","Data":"07a6c7680612f87ef21042e687bbe8011d744ff5d2760d38ff1de6aed12ebd1d"} Oct 30 17:00:56.885286 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:56.883765 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" podStartSLOduration=0.883716028 podStartE2EDuration="883.716028ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:56.882414781 +0000 UTC m=+2287.225088820" watchObservedRunningTime="2026-10-30 17:00:56.883716028 +0000 UTC m=+2287.226390064" Oct 30 17:00:57.299921 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:57.299880 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d"] Oct 30 17:00:57.351636 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:57.351603 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:00:57.353417 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:57.353392 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:00:57.353541 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:57.353438 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" podUID="40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 17:00:58.352116 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:58.352084 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:00:58.352527 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:58.352138 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" podUID="40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 17:00:59.352003 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:59.351966 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:00:59.352190 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:00:59.352019 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" podUID="40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 17:01:00.351680 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:00.351637 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:01:00.351916 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:00.351714 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" podUID="40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 17:01:01.351921 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:01.351883 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:01:01.352319 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:01.351949 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" podUID="40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 17:01:02.351332 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:02.351301 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 17:01:02.351508 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:02.351371 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" podUID="40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 17:01:03.356614 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:03.356580 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:01:03.357125 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:03.357107 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:01:03.357864 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:03.357845 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:01:04.461575 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:04.461539 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r"] Oct 30 17:01:04.467503 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:04.467481 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" Oct 30 17:01:04.486467 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:04.486443 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r"] Oct 30 17:01:04.494529 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:04.494508 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-81733d\"" Oct 30 17:01:04.494646 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:04.494610 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-81733d-dockercfg-9cgz4\"" Oct 30 17:01:04.601628 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:04.601592 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a745a6-d568-4614-a1cf-2c2f20f3c550-plugins-config-volume\") pod \"payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r\" (UID: \"93a745a6-d568-4614-a1cf-2c2f20f3c550\") " pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" Oct 30 17:01:04.601968 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:04.601939 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlltt\" (UniqueName: \"kubernetes.io/projected/93a745a6-d568-4614-a1cf-2c2f20f3c550-kube-api-access-wlltt\") pod \"payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r\" (UID: \"93a745a6-d568-4614-a1cf-2c2f20f3c550\") " pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" Oct 30 17:01:04.703207 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:04.703172 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a745a6-d568-4614-a1cf-2c2f20f3c550-plugins-config-volume\") pod \"payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r\" (UID: \"93a745a6-d568-4614-a1cf-2c2f20f3c550\") " pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" Oct 30 17:01:04.703412 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:04.703377 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlltt\" (UniqueName: \"kubernetes.io/projected/93a745a6-d568-4614-a1cf-2c2f20f3c550-kube-api-access-wlltt\") pod \"payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r\" (UID: \"93a745a6-d568-4614-a1cf-2c2f20f3c550\") " pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" Oct 30 17:01:04.704178 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:04.704150 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a745a6-d568-4614-a1cf-2c2f20f3c550-plugins-config-volume\") pod \"payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r\" (UID: \"93a745a6-d568-4614-a1cf-2c2f20f3c550\") " pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" Oct 30 17:01:04.739921 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:04.739894 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlltt\" (UniqueName: \"kubernetes.io/projected/93a745a6-d568-4614-a1cf-2c2f20f3c550-kube-api-access-wlltt\") pod \"payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r\" (UID: \"93a745a6-d568-4614-a1cf-2c2f20f3c550\") " pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" Oct 30 17:01:04.792743 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:04.783412 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" Oct 30 17:01:05.147738 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:05.147695 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r"] Oct 30 17:01:05.149529 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:01:05.149500 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a745a6_d568_4614_a1cf_2c2f20f3c550.slice/crio-24ac873972e19599ddcb2515a70b36a7253a8ff484056a17e68e3733fecfe3f7 WatchSource:0}: Error finding container 24ac873972e19599ddcb2515a70b36a7253a8ff484056a17e68e3733fecfe3f7: Status 404 returned error can't find the container with id 24ac873972e19599ddcb2515a70b36a7253a8ff484056a17e68e3733fecfe3f7 Oct 30 17:01:05.912432 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:05.912382 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" event={"ID":"93a745a6-d568-4614-a1cf-2c2f20f3c550","Type":"ContainerStarted","Data":"033d9929c193287147d22283560d74def7dd04f1d8dd62162ea712bf395b8410"} Oct 30 17:01:05.912913 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:05.912446 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" event={"ID":"93a745a6-d568-4614-a1cf-2c2f20f3c550","Type":"ContainerStarted","Data":"24ac873972e19599ddcb2515a70b36a7253a8ff484056a17e68e3733fecfe3f7"} Oct 30 17:01:05.912913 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:05.912472 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" Oct 30 17:01:05.969321 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:05.969269 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" podStartSLOduration=1.9692539679999999 podStartE2EDuration="1.969253968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:05.953937462 +0000 UTC m=+2296.296611524" watchObservedRunningTime="2026-10-30 17:01:05.969253968 +0000 UTC m=+2296.311928033" Oct 30 17:01:06.820771 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:06.820734 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm"] Oct 30 17:01:06.826227 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:06.826206 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" Oct 30 17:01:06.852586 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:06.846875 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm"] Oct 30 17:01:06.854635 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:06.854607 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-abda73\"" Oct 30 17:01:06.877252 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:06.877214 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-abda73-dockercfg-hw9d4\"" Oct 30 17:01:06.977746 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:06.977013 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2810edfd-dbe1-4bd4-9060-cf8c9e054741-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm\" (UID: \"2810edfd-dbe1-4bd4-9060-cf8c9e054741\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" Oct 30 17:01:06.977746 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:06.977142 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64lzr\" (UniqueName: \"kubernetes.io/projected/2810edfd-dbe1-4bd4-9060-cf8c9e054741-kube-api-access-64lzr\") pod \"payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm\" (UID: \"2810edfd-dbe1-4bd4-9060-cf8c9e054741\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" Oct 30 17:01:07.078927 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:07.078844 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2810edfd-dbe1-4bd4-9060-cf8c9e054741-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm\" (UID: \"2810edfd-dbe1-4bd4-9060-cf8c9e054741\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" Oct 30 17:01:07.079124 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:07.078948 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64lzr\" (UniqueName: \"kubernetes.io/projected/2810edfd-dbe1-4bd4-9060-cf8c9e054741-kube-api-access-64lzr\") pod \"payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm\" (UID: \"2810edfd-dbe1-4bd4-9060-cf8c9e054741\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" Oct 30 17:01:07.079801 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:07.079772 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2810edfd-dbe1-4bd4-9060-cf8c9e054741-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm\" (UID: \"2810edfd-dbe1-4bd4-9060-cf8c9e054741\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" Oct 30 17:01:07.103269 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:07.103238 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64lzr\" (UniqueName: \"kubernetes.io/projected/2810edfd-dbe1-4bd4-9060-cf8c9e054741-kube-api-access-64lzr\") pod \"payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm\" (UID: \"2810edfd-dbe1-4bd4-9060-cf8c9e054741\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" Oct 30 17:01:07.140916 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:07.140892 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" Oct 30 17:01:07.447973 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:07.447939 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm"] Oct 30 17:01:07.449369 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:01:07.449338 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2810edfd_dbe1_4bd4_9060_cf8c9e054741.slice/crio-56d37b39aadd4a151a490429ce8aa257cd357fcdbb0846d2b6f73a07f95662c7 WatchSource:0}: Error finding container 56d37b39aadd4a151a490429ce8aa257cd357fcdbb0846d2b6f73a07f95662c7: Status 404 returned error can't find the container with id 56d37b39aadd4a151a490429ce8aa257cd357fcdbb0846d2b6f73a07f95662c7 Oct 30 17:01:07.930807 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:07.930771 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" event={"ID":"2810edfd-dbe1-4bd4-9060-cf8c9e054741","Type":"ContainerStarted","Data":"86f493c314a1ce4f30d5737a73148c12e474579855e613a9bd66b4caecc211f3"} Oct 30 17:01:07.931010 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:07.930825 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" event={"ID":"2810edfd-dbe1-4bd4-9060-cf8c9e054741","Type":"ContainerStarted","Data":"56d37b39aadd4a151a490429ce8aa257cd357fcdbb0846d2b6f73a07f95662c7"} Oct 30 17:01:07.931010 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:07.930848 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" Oct 30 17:01:07.976640 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:07.976589 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" podStartSLOduration=1.9765759109999999 podStartE2EDuration="1.976575911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:07.973694651 +0000 UTC m=+2298.316368689" watchObservedRunningTime="2026-10-30 17:01:07.976575911 +0000 UTC m=+2298.319249948" Oct 30 17:01:10.501165 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.501121 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" podUID="fc1a58ac-4a05-4766-b212-7eff1c8c0e7f" containerName="main" containerID="cri-o://db9b896421464da50665563bfc4161046ba77b188c0a974f7137cf091c42b37d" gracePeriod=45 Oct 30 17:01:10.793122 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.793096 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 17:01:10.852193 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.852162 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-home\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-home\") pod \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " Oct 30 17:01:10.852383 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.852277 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-tls-certs\") pod \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " Oct 30 17:01:10.852383 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.852348 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-kube-api-access-htwxd\" (UniqueName: \"kubernetes.io/projected/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-kube-api-access-htwxd\") pod \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " Oct 30 17:01:10.852704 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.852398 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-model-cache\") pod \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " Oct 30 17:01:10.852704 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.852446 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-tmp-dir\") pod \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " Oct 30 17:01:10.852704 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.852538 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-dshm\") pod \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\" (UID: \"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f\") " Oct 30 17:01:10.852900 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.852829 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-model-cache" pod "fc1a58ac-4a05-4766-b212-7eff1c8c0e7f" (UID: "fc1a58ac-4a05-4766-b212-7eff1c8c0e7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:10.853009 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.852981 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-tmp-dir" pod "fc1a58ac-4a05-4766-b212-7eff1c8c0e7f" (UID: "fc1a58ac-4a05-4766-b212-7eff1c8c0e7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:10.853237 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.853013 4861 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:01:10.853237 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.853053 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-home" pod "fc1a58ac-4a05-4766-b212-7eff1c8c0e7f" (UID: "fc1a58ac-4a05-4766-b212-7eff1c8c0e7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:10.854880 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.854854 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-tls-certs" pod "fc1a58ac-4a05-4766-b212-7eff1c8c0e7f" (UID: "fc1a58ac-4a05-4766-b212-7eff1c8c0e7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:10.855078 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.855062 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-dshm" pod "fc1a58ac-4a05-4766-b212-7eff1c8c0e7f" (UID: "fc1a58ac-4a05-4766-b212-7eff1c8c0e7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:10.855414 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.855398 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-kube-api-access-htwxd" pod "fc1a58ac-4a05-4766-b212-7eff1c8c0e7f" (UID: "fc1a58ac-4a05-4766-b212-7eff1c8c0e7f"). InnerVolumeSpecName "kube-api-access-htwxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:10.953983 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.953953 4861 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:01:10.953983 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.953976 4861 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:01:10.954164 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.953991 4861 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:01:10.954164 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.954002 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htwxd\" (UniqueName: \"kubernetes.io/projected/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-kube-api-access-htwxd\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:01:10.954164 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.954010 4861 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:01:10.954770 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.954714 4861 generic.go:356] "Generic (PLEG): container finished" podID="fc1a58ac-4a05-4766-b212-7eff1c8c0e7f" containerID="db9b896421464da50665563bfc4161046ba77b188c0a974f7137cf091c42b37d" exitCode=0 Oct 30 17:01:10.954880 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.954774 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" event={"ID":"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f","Type":"ContainerDied","Data":"db9b896421464da50665563bfc4161046ba77b188c0a974f7137cf091c42b37d"} Oct 30 17:01:10.954880 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.954802 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" Oct 30 17:01:10.954880 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.954829 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq" event={"ID":"fc1a58ac-4a05-4766-b212-7eff1c8c0e7f","Type":"ContainerDied","Data":"03fb80be8ebe87bbfc70c1cc420f5568bad319bc547c99233f5eda1cbe654094"} Oct 30 17:01:10.954880 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.954854 4861 scope.go:122] "RemoveContainer" containerID="db9b896421464da50665563bfc4161046ba77b188c0a974f7137cf091c42b37d" Oct 30 17:01:10.967054 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.967028 4861 scope.go:122] "RemoveContainer" containerID="db9b896421464da50665563bfc4161046ba77b188c0a974f7137cf091c42b37d" Oct 30 17:01:10.967371 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:01:10.967351 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db9b896421464da50665563bfc4161046ba77b188c0a974f7137cf091c42b37d\": container with ID starting with db9b896421464da50665563bfc4161046ba77b188c0a974f7137cf091c42b37d not found: ID does not exist" containerID="db9b896421464da50665563bfc4161046ba77b188c0a974f7137cf091c42b37d" Oct 30 17:01:10.967446 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:10.967382 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db9b896421464da50665563bfc4161046ba77b188c0a974f7137cf091c42b37d"} err="failed to get container status \"db9b896421464da50665563bfc4161046ba77b188c0a974f7137cf091c42b37d\": rpc error: code = NotFound desc = could not find container \"db9b896421464da50665563bfc4161046ba77b188c0a974f7137cf091c42b37d\": container with ID starting with db9b896421464da50665563bfc4161046ba77b188c0a974f7137cf091c42b37d not found: ID does not exist" Oct 30 17:01:11.013224 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:11.013129 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq"] Oct 30 17:01:11.016389 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:11.016361 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-799c76/auth-test-model-799c76-kserve-5c48df4649-9d8gq"] Oct 30 17:01:11.808369 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:11.808336 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc1a58ac-4a05-4766-b212-7eff1c8c0e7f" path="/var/lib/kubelet/pods/fc1a58ac-4a05-4766-b212-7eff1c8c0e7f/volumes" Oct 30 17:01:12.350390 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.350351 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" podUID="7bab337b-715f-4094-840e-75f1fba95a21" containerName="main" containerID="cri-o://38f2f8052ceff40ddcf044ae18729dd3f76491b3438ee3e388f274198f0bfe85" gracePeriod=45 Oct 30 17:01:12.636684 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.636656 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 17:01:12.676087 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.676061 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-dshm\") pod \"7bab337b-715f-4094-840e-75f1fba95a21\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " Oct 30 17:01:12.676217 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.676143 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bab337b-715f-4094-840e-75f1fba95a21-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bab337b-715f-4094-840e-75f1fba95a21-tls-certs\") pod \"7bab337b-715f-4094-840e-75f1fba95a21\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " Oct 30 17:01:12.676217 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.676173 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-home\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-home\") pod \"7bab337b-715f-4094-840e-75f1fba95a21\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " Oct 30 17:01:12.676338 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.676220 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-tmp-dir\") pod \"7bab337b-715f-4094-840e-75f1fba95a21\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " Oct 30 17:01:12.676338 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.676292 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bab337b-715f-4094-840e-75f1fba95a21-kube-api-access-mxs6b\" (UniqueName: \"kubernetes.io/projected/7bab337b-715f-4094-840e-75f1fba95a21-kube-api-access-mxs6b\") pod \"7bab337b-715f-4094-840e-75f1fba95a21\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " Oct 30 17:01:12.676455 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.676372 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-model-cache\") pod \"7bab337b-715f-4094-840e-75f1fba95a21\" (UID: \"7bab337b-715f-4094-840e-75f1fba95a21\") " Oct 30 17:01:12.676455 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.676439 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-tmp-dir" pod "7bab337b-715f-4094-840e-75f1fba95a21" (UID: "7bab337b-715f-4094-840e-75f1fba95a21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:12.676552 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.676476 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-home" pod "7bab337b-715f-4094-840e-75f1fba95a21" (UID: "7bab337b-715f-4094-840e-75f1fba95a21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:12.676829 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.676709 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-model-cache" pod "7bab337b-715f-4094-840e-75f1fba95a21" (UID: "7bab337b-715f-4094-840e-75f1fba95a21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:12.676949 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.676921 4861 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:01:12.676949 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.676941 4861 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:01:12.677057 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.676953 4861 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:01:12.679267 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.679207 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bab337b-715f-4094-840e-75f1fba95a21-tls-certs" pod "7bab337b-715f-4094-840e-75f1fba95a21" (UID: "7bab337b-715f-4094-840e-75f1fba95a21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:12.679390 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.679366 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bab337b-715f-4094-840e-75f1fba95a21-kube-api-access-mxs6b" pod "7bab337b-715f-4094-840e-75f1fba95a21" (UID: "7bab337b-715f-4094-840e-75f1fba95a21"). InnerVolumeSpecName "kube-api-access-mxs6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:12.679461 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.679371 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-dshm" pod "7bab337b-715f-4094-840e-75f1fba95a21" (UID: "7bab337b-715f-4094-840e-75f1fba95a21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:12.778610 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.778583 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxs6b\" (UniqueName: \"kubernetes.io/projected/7bab337b-715f-4094-840e-75f1fba95a21-kube-api-access-mxs6b\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:01:12.778610 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.778611 4861 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bab337b-715f-4094-840e-75f1fba95a21-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:01:12.778810 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.778626 4861 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bab337b-715f-4094-840e-75f1fba95a21-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:01:12.996748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.991556 4861 generic.go:356] "Generic (PLEG): container finished" podID="7bab337b-715f-4094-840e-75f1fba95a21" containerID="38f2f8052ceff40ddcf044ae18729dd3f76491b3438ee3e388f274198f0bfe85" exitCode=0 Oct 30 17:01:12.996748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.991633 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" event={"ID":"7bab337b-715f-4094-840e-75f1fba95a21","Type":"ContainerDied","Data":"38f2f8052ceff40ddcf044ae18729dd3f76491b3438ee3e388f274198f0bfe85"} Oct 30 17:01:12.996748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.991667 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" event={"ID":"7bab337b-715f-4094-840e-75f1fba95a21","Type":"ContainerDied","Data":"d68591efeac77175a958e5eb34bda34f23908b616a8ae699f28adb49e3a64597"} Oct 30 17:01:12.996748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.991701 4861 scope.go:122] "RemoveContainer" containerID="38f2f8052ceff40ddcf044ae18729dd3f76491b3438ee3e388f274198f0bfe85" Oct 30 17:01:12.996748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:12.991869 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d" Oct 30 17:01:13.021961 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:13.021175 4861 scope.go:122] "RemoveContainer" containerID="38f2f8052ceff40ddcf044ae18729dd3f76491b3438ee3e388f274198f0bfe85" Oct 30 17:01:13.021961 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:01:13.021535 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f2f8052ceff40ddcf044ae18729dd3f76491b3438ee3e388f274198f0bfe85\": container with ID starting with 38f2f8052ceff40ddcf044ae18729dd3f76491b3438ee3e388f274198f0bfe85 not found: ID does not exist" containerID="38f2f8052ceff40ddcf044ae18729dd3f76491b3438ee3e388f274198f0bfe85" Oct 30 17:01:13.021961 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:13.021578 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f2f8052ceff40ddcf044ae18729dd3f76491b3438ee3e388f274198f0bfe85"} err="failed to get container status \"38f2f8052ceff40ddcf044ae18729dd3f76491b3438ee3e388f274198f0bfe85\": rpc error: code = NotFound desc = could not find container \"38f2f8052ceff40ddcf044ae18729dd3f76491b3438ee3e388f274198f0bfe85\": container with ID starting with 38f2f8052ceff40ddcf044ae18729dd3f76491b3438ee3e388f274198f0bfe85 not found: ID does not exist" Oct 30 17:01:13.052835 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:13.052801 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d"] Oct 30 17:01:13.055969 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:13.055944 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-06e7f6/auth-test-model-06e7f6-kserve-84c5b44965-h459d"] Oct 30 17:01:13.807876 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:13.807845 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bab337b-715f-4094-840e-75f1fba95a21" path="/var/lib/kubelet/pods/7bab337b-715f-4094-840e-75f1fba95a21/volumes" Oct 30 17:01:16.925353 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:16.925324 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" Oct 30 17:01:18.941051 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:18.941022 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" Oct 30 17:01:20.296431 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:01:20.296391 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:01:22.660267 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.660233 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj"] Oct 30 17:01:22.661030 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.661012 4861 cpu_manager.go:414] "containerMap: removing container" podUID="fc1a58ac-4a05-4766-b212-7eff1c8c0e7f" containerName="main" Oct 30 17:01:22.661030 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.661031 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1a58ac-4a05-4766-b212-7eff1c8c0e7f" containerName="main" Oct 30 17:01:22.661154 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.661050 4861 cpu_manager.go:414] "containerMap: removing container" podUID="7bab337b-715f-4094-840e-75f1fba95a21" containerName="main" Oct 30 17:01:22.661154 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.661055 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bab337b-715f-4094-840e-75f1fba95a21" containerName="main" Oct 30 17:01:22.661154 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.661150 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bab337b-715f-4094-840e-75f1fba95a21" containerName="main" Oct 30 17:01:22.661258 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.661168 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1a58ac-4a05-4766-b212-7eff1c8c0e7f" containerName="main" Oct 30 17:01:22.664879 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.664862 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" Oct 30 17:01:22.684604 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.684579 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj"] Oct 30 17:01:22.694589 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.694569 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-13c495\"" Oct 30 17:01:22.695074 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.695058 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-13c495-dockercfg-jsnjc\"" Oct 30 17:01:22.768974 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.768934 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qswrn\" (UniqueName: \"kubernetes.io/projected/40c65962-30bc-4ea9-b938-4a4917e875cc-kube-api-access-qswrn\") pod \"payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj\" (UID: \"40c65962-30bc-4ea9-b938-4a4917e875cc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" Oct 30 17:01:22.769347 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.769318 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c65962-30bc-4ea9-b938-4a4917e875cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj\" (UID: \"40c65962-30bc-4ea9-b938-4a4917e875cc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" Oct 30 17:01:22.843787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.828716 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp"] Oct 30 17:01:22.845884 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.845854 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp"] Oct 30 17:01:22.846188 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.846170 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" Oct 30 17:01:22.870929 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.870903 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qswrn\" (UniqueName: \"kubernetes.io/projected/40c65962-30bc-4ea9-b938-4a4917e875cc-kube-api-access-qswrn\") pod \"payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj\" (UID: \"40c65962-30bc-4ea9-b938-4a4917e875cc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" Oct 30 17:01:22.871071 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.870957 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c65962-30bc-4ea9-b938-4a4917e875cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj\" (UID: \"40c65962-30bc-4ea9-b938-4a4917e875cc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" Oct 30 17:01:22.871643 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.871622 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c65962-30bc-4ea9-b938-4a4917e875cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj\" (UID: \"40c65962-30bc-4ea9-b938-4a4917e875cc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" Oct 30 17:01:22.899041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.899003 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qswrn\" (UniqueName: \"kubernetes.io/projected/40c65962-30bc-4ea9-b938-4a4917e875cc-kube-api-access-qswrn\") pod \"payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj\" (UID: \"40c65962-30bc-4ea9-b938-4a4917e875cc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" Oct 30 17:01:22.971934 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.971906 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9jr5\" (UniqueName: \"kubernetes.io/projected/ce164706-e748-479e-a1a6-7977e983bc9e-kube-api-access-d9jr5\") pod \"payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp\" (UID: \"ce164706-e748-479e-a1a6-7977e983bc9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" Oct 30 17:01:22.972105 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.972085 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce164706-e748-479e-a1a6-7977e983bc9e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp\" (UID: \"ce164706-e748-479e-a1a6-7977e983bc9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" Oct 30 17:01:22.979816 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:22.979792 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" Oct 30 17:01:23.076042 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:23.073899 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce164706-e748-479e-a1a6-7977e983bc9e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp\" (UID: \"ce164706-e748-479e-a1a6-7977e983bc9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" Oct 30 17:01:23.076042 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:23.074084 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9jr5\" (UniqueName: \"kubernetes.io/projected/ce164706-e748-479e-a1a6-7977e983bc9e-kube-api-access-d9jr5\") pod \"payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp\" (UID: \"ce164706-e748-479e-a1a6-7977e983bc9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" Oct 30 17:01:23.076312 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:23.076288 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce164706-e748-479e-a1a6-7977e983bc9e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp\" (UID: \"ce164706-e748-479e-a1a6-7977e983bc9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" Oct 30 17:01:23.100962 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:23.100937 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9jr5\" (UniqueName: \"kubernetes.io/projected/ce164706-e748-479e-a1a6-7977e983bc9e-kube-api-access-d9jr5\") pod \"payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp\" (UID: \"ce164706-e748-479e-a1a6-7977e983bc9e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" Oct 30 17:01:23.170856 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:23.170829 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" Oct 30 17:01:23.311820 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:23.311775 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj"] Oct 30 17:01:23.315457 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:01:23.315415 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c65962_30bc_4ea9_b938_4a4917e875cc.slice/crio-5b17327c9c6f8a9d9d3a40cfbc7ee465d3076c562627a61fd7b181668034f809 WatchSource:0}: Error finding container 5b17327c9c6f8a9d9d3a40cfbc7ee465d3076c562627a61fd7b181668034f809: Status 404 returned error can't find the container with id 5b17327c9c6f8a9d9d3a40cfbc7ee465d3076c562627a61fd7b181668034f809 Oct 30 17:01:23.477538 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:23.477513 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp"] Oct 30 17:01:23.478540 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:01:23.478512 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce164706_e748_479e_a1a6_7977e983bc9e.slice/crio-4b18ac22e4fd89fc4df85323b23c4fa581861426fabbf3372481a9242eaafa88 WatchSource:0}: Error finding container 4b18ac22e4fd89fc4df85323b23c4fa581861426fabbf3372481a9242eaafa88: Status 404 returned error can't find the container with id 4b18ac22e4fd89fc4df85323b23c4fa581861426fabbf3372481a9242eaafa88 Oct 30 17:01:24.072653 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:24.072609 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" event={"ID":"40c65962-30bc-4ea9-b938-4a4917e875cc","Type":"ContainerStarted","Data":"87eccaefa07e400cd38b7ac2c6833f5c30115424fddc38a412fe17abfb84133f"} Oct 30 17:01:24.073150 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:24.072677 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" event={"ID":"40c65962-30bc-4ea9-b938-4a4917e875cc","Type":"ContainerStarted","Data":"5b17327c9c6f8a9d9d3a40cfbc7ee465d3076c562627a61fd7b181668034f809"} Oct 30 17:01:24.073150 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:24.072848 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" Oct 30 17:01:24.074300 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:24.074272 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" event={"ID":"ce164706-e748-479e-a1a6-7977e983bc9e","Type":"ContainerStarted","Data":"81b274cea3578b2b0f44f15cb1034c7ad6fba6244210eb9e19b3701bd7a96f9f"} Oct 30 17:01:24.074448 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:24.074317 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" event={"ID":"ce164706-e748-479e-a1a6-7977e983bc9e","Type":"ContainerStarted","Data":"4b18ac22e4fd89fc4df85323b23c4fa581861426fabbf3372481a9242eaafa88"} Oct 30 17:01:24.074448 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:24.074346 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" Oct 30 17:01:24.119974 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:24.119920 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" podStartSLOduration=2.119908568 podStartE2EDuration="2.119908568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:24.118475746 +0000 UTC m=+2314.461149785" watchObservedRunningTime="2026-10-30 17:01:24.119908568 +0000 UTC m=+2314.462582606" Oct 30 17:01:24.166570 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:24.166528 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" podStartSLOduration=2.166517227 podStartE2EDuration="2.166517227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:24.163827836 +0000 UTC m=+2314.506501891" watchObservedRunningTime="2026-10-30 17:01:24.166517227 +0000 UTC m=+2314.509191297" Oct 30 17:01:35.084261 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:35.084225 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" Oct 30 17:01:35.084806 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:35.084474 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" Oct 30 17:01:37.234264 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.234227 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq"] Oct 30 17:01:37.240324 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.240306 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:01:37.261256 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.261217 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq"] Oct 30 17:01:37.270115 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.270089 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-81733d-serving-cert\"" Oct 30 17:01:37.366793 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.366765 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgq5w\" (UniqueName: \"kubernetes.io/projected/0697e319-8f24-4056-a1a7-e9782794dd7f-kube-api-access-dgq5w\") pod \"maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq\" (UID: \"0697e319-8f24-4056-a1a7-e9782794dd7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:01:37.366956 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.366799 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0697e319-8f24-4056-a1a7-e9782794dd7f-maas-api-tls\") pod \"maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq\" (UID: \"0697e319-8f24-4056-a1a7-e9782794dd7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:01:37.366956 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.366897 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0697e319-8f24-4056-a1a7-e9782794dd7f-metrics-certs\") pod \"maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq\" (UID: \"0697e319-8f24-4056-a1a7-e9782794dd7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:01:37.367087 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.367066 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0697e319-8f24-4056-a1a7-e9782794dd7f-openshift-service-ca\") pod \"maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq\" (UID: \"0697e319-8f24-4056-a1a7-e9782794dd7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:01:37.468792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.468751 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgq5w\" (UniqueName: \"kubernetes.io/projected/0697e319-8f24-4056-a1a7-e9782794dd7f-kube-api-access-dgq5w\") pod \"maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq\" (UID: \"0697e319-8f24-4056-a1a7-e9782794dd7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:01:37.469013 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.468805 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0697e319-8f24-4056-a1a7-e9782794dd7f-maas-api-tls\") pod \"maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq\" (UID: \"0697e319-8f24-4056-a1a7-e9782794dd7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:01:37.469013 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.468890 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0697e319-8f24-4056-a1a7-e9782794dd7f-metrics-certs\") pod \"maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq\" (UID: \"0697e319-8f24-4056-a1a7-e9782794dd7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:01:37.469153 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.469033 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0697e319-8f24-4056-a1a7-e9782794dd7f-openshift-service-ca\") pod \"maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq\" (UID: \"0697e319-8f24-4056-a1a7-e9782794dd7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:01:37.469925 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.469895 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0697e319-8f24-4056-a1a7-e9782794dd7f-openshift-service-ca\") pod \"maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq\" (UID: \"0697e319-8f24-4056-a1a7-e9782794dd7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:01:37.481762 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.479672 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0697e319-8f24-4056-a1a7-e9782794dd7f-maas-api-tls\") pod \"maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq\" (UID: \"0697e319-8f24-4056-a1a7-e9782794dd7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:01:37.485800 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.483226 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0697e319-8f24-4056-a1a7-e9782794dd7f-metrics-certs\") pod \"maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq\" (UID: \"0697e319-8f24-4056-a1a7-e9782794dd7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:01:37.510100 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.509998 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgq5w\" (UniqueName: \"kubernetes.io/projected/0697e319-8f24-4056-a1a7-e9782794dd7f-kube-api-access-dgq5w\") pod \"maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq\" (UID: \"0697e319-8f24-4056-a1a7-e9782794dd7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:01:37.556006 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.555973 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:01:37.865642 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:37.865610 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq"] Oct 30 17:01:37.866320 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:01:37.866287 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0697e319_8f24_4056_a1a7_e9782794dd7f.slice/crio-4e3fc37819f43e5d34bbe591e31c9c073e2a7a1ec38eb3f3c5c58b6304072972 WatchSource:0}: Error finding container 4e3fc37819f43e5d34bbe591e31c9c073e2a7a1ec38eb3f3c5c58b6304072972: Status 404 returned error can't find the container with id 4e3fc37819f43e5d34bbe591e31c9c073e2a7a1ec38eb3f3c5c58b6304072972 Oct 30 17:01:38.175079 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:38.175004 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" event={"ID":"0697e319-8f24-4056-a1a7-e9782794dd7f","Type":"ContainerStarted","Data":"4e3fc37819f43e5d34bbe591e31c9c073e2a7a1ec38eb3f3c5c58b6304072972"} Oct 30 17:01:39.183802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:39.183754 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" event={"ID":"0697e319-8f24-4056-a1a7-e9782794dd7f","Type":"ContainerStarted","Data":"72b323ca724be6344e2310742aa6da25140949c6469cb9498fdeb48a1b5f2dc8"} Oct 30 17:01:39.184254 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:39.183945 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:01:39.231554 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:39.231503 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" podStartSLOduration=1.902464634 podStartE2EDuration="2.231490399s" totalImagesPullingTime="329.025765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:39.231369227 +0000 UTC m=+2329.574043266" watchObservedRunningTime="2026-10-30 17:01:39.231490399 +0000 UTC m=+2329.574164437" Oct 30 17:01:45.196904 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:45.196875 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:01:50.295649 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:01:50.295609 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:01:50.919855 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:50.919823 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl"] Oct 30 17:01:50.924738 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:50.924699 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" Oct 30 17:01:50.953593 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:50.953545 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-5c6b3c\"" Oct 30 17:01:50.953963 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:50.953945 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-5c6b3c-dockercfg-f77zp\"" Oct 30 17:01:50.958337 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:50.958317 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl"] Oct 30 17:01:51.031747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.029097 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb"] Oct 30 17:01:51.047747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.042387 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" Oct 30 17:01:51.047747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.045375 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb"] Oct 30 17:01:51.115285 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.115246 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf4ec76-a9e6-487b-a504-67adbc815d25-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl\" (UID: \"caf4ec76-a9e6-487b-a504-67adbc815d25\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" Oct 30 17:01:51.116463 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.115403 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv7rm\" (UniqueName: \"kubernetes.io/projected/caf4ec76-a9e6-487b-a504-67adbc815d25-kube-api-access-lv7rm\") pod \"payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl\" (UID: \"caf4ec76-a9e6-487b-a504-67adbc815d25\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" Oct 30 17:01:51.216610 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.216508 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf4ec76-a9e6-487b-a504-67adbc815d25-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl\" (UID: \"caf4ec76-a9e6-487b-a504-67adbc815d25\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" Oct 30 17:01:51.216836 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.216661 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv7rm\" (UniqueName: \"kubernetes.io/projected/caf4ec76-a9e6-487b-a504-67adbc815d25-kube-api-access-lv7rm\") pod \"payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl\" (UID: \"caf4ec76-a9e6-487b-a504-67adbc815d25\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" Oct 30 17:01:51.216836 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.216752 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d980004b-323c-4ebf-be4e-d0356007ba80-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb\" (UID: \"d980004b-323c-4ebf-be4e-d0356007ba80\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" Oct 30 17:01:51.216836 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.216809 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42x4m\" (UniqueName: \"kubernetes.io/projected/d980004b-323c-4ebf-be4e-d0356007ba80-kube-api-access-42x4m\") pod \"payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb\" (UID: \"d980004b-323c-4ebf-be4e-d0356007ba80\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" Oct 30 17:01:51.217359 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.217337 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf4ec76-a9e6-487b-a504-67adbc815d25-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl\" (UID: \"caf4ec76-a9e6-487b-a504-67adbc815d25\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" Oct 30 17:01:51.240817 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.240794 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv7rm\" (UniqueName: \"kubernetes.io/projected/caf4ec76-a9e6-487b-a504-67adbc815d25-kube-api-access-lv7rm\") pod \"payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl\" (UID: \"caf4ec76-a9e6-487b-a504-67adbc815d25\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" Oct 30 17:01:51.258707 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.258681 4861 scope.go:122] "RemoveContainer" containerID="efbbb811180ae2b533106299d8b5a5cf1ca3f03afcf8b167a2a2f6d1ee02974e" Oct 30 17:01:51.318337 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.318304 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42x4m\" (UniqueName: \"kubernetes.io/projected/d980004b-323c-4ebf-be4e-d0356007ba80-kube-api-access-42x4m\") pod \"payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb\" (UID: \"d980004b-323c-4ebf-be4e-d0356007ba80\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" Oct 30 17:01:51.318713 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.318588 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d980004b-323c-4ebf-be4e-d0356007ba80-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb\" (UID: \"d980004b-323c-4ebf-be4e-d0356007ba80\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" Oct 30 17:01:51.319252 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.319231 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d980004b-323c-4ebf-be4e-d0356007ba80-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb\" (UID: \"d980004b-323c-4ebf-be4e-d0356007ba80\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" Oct 30 17:01:51.342486 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.342461 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42x4m\" (UniqueName: \"kubernetes.io/projected/d980004b-323c-4ebf-be4e-d0356007ba80-kube-api-access-42x4m\") pod \"payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb\" (UID: \"d980004b-323c-4ebf-be4e-d0356007ba80\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" Oct 30 17:01:51.372119 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.372098 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" Oct 30 17:01:51.539066 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.538995 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" Oct 30 17:01:51.691248 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.691217 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb"] Oct 30 17:01:51.691623 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:01:51.691604 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd980004b_323c_4ebf_be4e_d0356007ba80.slice/crio-94cbd0b3142dd5d31b82f9320539235be7dd81d695bfe3155205b7671115b6f3 WatchSource:0}: Error finding container 94cbd0b3142dd5d31b82f9320539235be7dd81d695bfe3155205b7671115b6f3: Status 404 returned error can't find the container with id 94cbd0b3142dd5d31b82f9320539235be7dd81d695bfe3155205b7671115b6f3 Oct 30 17:01:51.856971 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:51.856935 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl"] Oct 30 17:01:51.857215 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:01:51.857194 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf4ec76_a9e6_487b_a504_67adbc815d25.slice/crio-55d890661eebee9340c476ac788886f8b58807e794b42e7781f1806075a02547 WatchSource:0}: Error finding container 55d890661eebee9340c476ac788886f8b58807e794b42e7781f1806075a02547: Status 404 returned error can't find the container with id 55d890661eebee9340c476ac788886f8b58807e794b42e7781f1806075a02547 Oct 30 17:01:52.286947 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:52.286907 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" event={"ID":"caf4ec76-a9e6-487b-a504-67adbc815d25","Type":"ContainerStarted","Data":"c62a23ed7ff52a23471a60af98d37483628cb6e40b811e3d464f37d9746f0ea1"} Oct 30 17:01:52.287147 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:52.287109 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" Oct 30 17:01:52.287224 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:52.287152 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" event={"ID":"caf4ec76-a9e6-487b-a504-67adbc815d25","Type":"ContainerStarted","Data":"55d890661eebee9340c476ac788886f8b58807e794b42e7781f1806075a02547"} Oct 30 17:01:52.288550 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:52.288525 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" event={"ID":"d980004b-323c-4ebf-be4e-d0356007ba80","Type":"ContainerStarted","Data":"b3b23f61a23c3c18ee7db5db622be86c85b9c493aa4d1e3ad9f3f669f3bdc90d"} Oct 30 17:01:52.288678 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:52.288569 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" event={"ID":"d980004b-323c-4ebf-be4e-d0356007ba80","Type":"ContainerStarted","Data":"94cbd0b3142dd5d31b82f9320539235be7dd81d695bfe3155205b7671115b6f3"} Oct 30 17:01:52.288678 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:52.288595 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" Oct 30 17:01:52.332534 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:52.332471 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" podStartSLOduration=2.332454556 podStartE2EDuration="2.332454556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:52.328925519 +0000 UTC m=+2342.671599551" watchObservedRunningTime="2026-10-30 17:01:52.332454556 +0000 UTC m=+2342.675128595" Oct 30 17:01:52.370706 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:01:52.370662 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" podStartSLOduration=2.370649481 podStartE2EDuration="2.370649481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:52.370233889 +0000 UTC m=+2342.712907929" watchObservedRunningTime="2026-10-30 17:01:52.370649481 +0000 UTC m=+2342.713323519" Oct 30 17:02:02.407985 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.407955 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7"] Oct 30 17:02:02.414223 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.414196 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" Oct 30 17:02:02.439701 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.439665 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-5758b7\"" Oct 30 17:02:02.443393 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.443370 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7"] Oct 30 17:02:02.452553 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.452530 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-5758b7-dockercfg-kqtqk\"" Oct 30 17:02:02.457900 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.457879 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7"] Oct 30 17:02:02.462383 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.462366 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" Oct 30 17:02:02.487341 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.487314 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7"] Oct 30 17:02:02.493481 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.493431 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956234cd-9c32-48f5-853f-7646454ab761-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7\" (UID: \"956234cd-9c32-48f5-853f-7646454ab761\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" Oct 30 17:02:02.494074 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.494033 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxbqx\" (UniqueName: \"kubernetes.io/projected/956234cd-9c32-48f5-853f-7646454ab761-kube-api-access-fxbqx\") pod \"payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7\" (UID: \"956234cd-9c32-48f5-853f-7646454ab761\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" Oct 30 17:02:02.597197 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.596990 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956234cd-9c32-48f5-853f-7646454ab761-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7\" (UID: \"956234cd-9c32-48f5-853f-7646454ab761\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" Oct 30 17:02:02.597397 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.597204 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0209596e-c5bc-4751-9864-0574cabf4197-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7\" (UID: \"0209596e-c5bc-4751-9864-0574cabf4197\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" Oct 30 17:02:02.597397 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.597282 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxbqx\" (UniqueName: \"kubernetes.io/projected/956234cd-9c32-48f5-853f-7646454ab761-kube-api-access-fxbqx\") pod \"payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7\" (UID: \"956234cd-9c32-48f5-853f-7646454ab761\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" Oct 30 17:02:02.597397 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.597344 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz5s4\" (UniqueName: \"kubernetes.io/projected/0209596e-c5bc-4751-9864-0574cabf4197-kube-api-access-kz5s4\") pod \"payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7\" (UID: \"0209596e-c5bc-4751-9864-0574cabf4197\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" Oct 30 17:02:02.599383 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.599356 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956234cd-9c32-48f5-853f-7646454ab761-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7\" (UID: \"956234cd-9c32-48f5-853f-7646454ab761\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" Oct 30 17:02:02.637765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.637195 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxbqx\" (UniqueName: \"kubernetes.io/projected/956234cd-9c32-48f5-853f-7646454ab761-kube-api-access-fxbqx\") pod \"payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7\" (UID: \"956234cd-9c32-48f5-853f-7646454ab761\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" Oct 30 17:02:02.698859 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.698778 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0209596e-c5bc-4751-9864-0574cabf4197-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7\" (UID: \"0209596e-c5bc-4751-9864-0574cabf4197\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" Oct 30 17:02:02.699027 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.698893 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz5s4\" (UniqueName: \"kubernetes.io/projected/0209596e-c5bc-4751-9864-0574cabf4197-kube-api-access-kz5s4\") pod \"payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7\" (UID: \"0209596e-c5bc-4751-9864-0574cabf4197\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" Oct 30 17:02:02.699524 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.699504 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0209596e-c5bc-4751-9864-0574cabf4197-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7\" (UID: \"0209596e-c5bc-4751-9864-0574cabf4197\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" Oct 30 17:02:02.726089 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.726052 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz5s4\" (UniqueName: \"kubernetes.io/projected/0209596e-c5bc-4751-9864-0574cabf4197-kube-api-access-kz5s4\") pod \"payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7\" (UID: \"0209596e-c5bc-4751-9864-0574cabf4197\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" Oct 30 17:02:02.732593 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.732568 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" Oct 30 17:02:02.786555 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:02.786520 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" Oct 30 17:02:03.044837 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:03.044811 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7"] Oct 30 17:02:03.045395 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:02:03.045371 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956234cd_9c32_48f5_853f_7646454ab761.slice/crio-ac1e19c4d11ee2190d82538ab53b000d91028d5d0cc02444a7fca851dd23474c WatchSource:0}: Error finding container ac1e19c4d11ee2190d82538ab53b000d91028d5d0cc02444a7fca851dd23474c: Status 404 returned error can't find the container with id ac1e19c4d11ee2190d82538ab53b000d91028d5d0cc02444a7fca851dd23474c Oct 30 17:02:03.146820 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:03.146795 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7"] Oct 30 17:02:03.147792 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:02:03.147766 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0209596e_c5bc_4751_9864_0574cabf4197.slice/crio-deeb2c1691a20edd9df7739697d731b16b5227d03c5b00236e3514156de3ef63 WatchSource:0}: Error finding container deeb2c1691a20edd9df7739697d731b16b5227d03c5b00236e3514156de3ef63: Status 404 returned error can't find the container with id deeb2c1691a20edd9df7739697d731b16b5227d03c5b00236e3514156de3ef63 Oct 30 17:02:03.299281 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:03.299202 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" Oct 30 17:02:03.299594 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:03.299575 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" Oct 30 17:02:03.397928 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:03.397592 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" event={"ID":"956234cd-9c32-48f5-853f-7646454ab761","Type":"ContainerStarted","Data":"fbe479ecc5cb0f01951ace6b555938febbf8cc08a215a5252ede772ecda9d6a9"} Oct 30 17:02:03.397928 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:03.397657 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" event={"ID":"956234cd-9c32-48f5-853f-7646454ab761","Type":"ContainerStarted","Data":"ac1e19c4d11ee2190d82538ab53b000d91028d5d0cc02444a7fca851dd23474c"} Oct 30 17:02:03.397928 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:03.397782 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" Oct 30 17:02:03.399897 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:03.399871 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" event={"ID":"0209596e-c5bc-4751-9864-0574cabf4197","Type":"ContainerStarted","Data":"9aa02aae4fc57ab7de21cbb3ab5622c166d22f635047f99a55bd8a97d6fc61d2"} Oct 30 17:02:03.400021 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:03.399912 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" event={"ID":"0209596e-c5bc-4751-9864-0574cabf4197","Type":"ContainerStarted","Data":"deeb2c1691a20edd9df7739697d731b16b5227d03c5b00236e3514156de3ef63"} Oct 30 17:02:03.400021 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:03.399949 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" Oct 30 17:02:03.443524 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:03.443474 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" podStartSLOduration=1.443461034 podStartE2EDuration="1.443461034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:03.443211227 +0000 UTC m=+2353.785885265" watchObservedRunningTime="2026-10-30 17:02:03.443461034 +0000 UTC m=+2353.786135071" Oct 30 17:02:03.491748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:03.491675 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" podStartSLOduration=1.491662895 podStartE2EDuration="1.491662895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:03.488252064 +0000 UTC m=+2353.830926102" watchObservedRunningTime="2026-10-30 17:02:03.491662895 +0000 UTC m=+2353.834336932" Oct 30 17:02:13.690377 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.690340 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq"] Oct 30 17:02:13.696809 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.696785 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.714146 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.714094 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-81733d-openshift-default-dockercfg-6hh6m\"" Oct 30 17:02:13.734900 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.734871 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq"] Oct 30 17:02:13.747717 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.747679 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8682959-6953-4597-9c34-200732698947-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.747909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.747853 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8682959-6953-4597-9c34-200732698947-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.748028 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.747985 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzjxc\" (UniqueName: \"kubernetes.io/projected/c8682959-6953-4597-9c34-200732698947-kube-api-access-tzjxc\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.748200 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.748057 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.748200 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.748098 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8682959-6953-4597-9c34-200732698947-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.748200 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.748128 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.748200 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.748154 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.748200 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.748186 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.748559 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.748224 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.849242 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.849209 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8682959-6953-4597-9c34-200732698947-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.849420 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.849311 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzjxc\" (UniqueName: \"kubernetes.io/projected/c8682959-6953-4597-9c34-200732698947-kube-api-access-tzjxc\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.849420 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.849356 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.849420 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.849395 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8682959-6953-4597-9c34-200732698947-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.849604 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.849426 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.849604 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.849454 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.849604 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.849489 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.849604 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.849528 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.849604 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.849589 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8682959-6953-4597-9c34-200732698947-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.849926 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.849852 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.850447 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.850118 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.850592 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.850565 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.850752 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.850644 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.850843 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.850795 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8682959-6953-4597-9c34-200732698947-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.852129 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.852107 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.852372 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.852355 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8682959-6953-4597-9c34-200732698947-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.875125 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.875105 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8682959-6953-4597-9c34-200732698947-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:13.875286 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:13.875269 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzjxc\" (UniqueName: \"kubernetes.io/projected/c8682959-6953-4597-9c34-200732698947-kube-api-access-tzjxc\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-ckwrq\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:14.014601 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:14.014575 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:14.341458 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:14.341426 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq"] Oct 30 17:02:14.342023 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:02:14.341997 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8682959_6953_4597_9c34_200732698947.slice/crio-837efdfdc8dc7f1b38914a9f78f524e3b43d9b14fcb059c384f701b1842cc98f WatchSource:0}: Error finding container 837efdfdc8dc7f1b38914a9f78f524e3b43d9b14fcb059c384f701b1842cc98f: Status 404 returned error can't find the container with id 837efdfdc8dc7f1b38914a9f78f524e3b43d9b14fcb059c384f701b1842cc98f Oct 30 17:02:14.355410 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:14.355363 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:02:14.355516 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:14.355452 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:02:14.355516 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:14.355482 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:02:14.411387 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:14.411363 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" Oct 30 17:02:14.411511 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:14.411437 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" Oct 30 17:02:14.487871 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:14.487048 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" event={"ID":"c8682959-6953-4597-9c34-200732698947","Type":"ContainerStarted","Data":"c8ab7e8f793235e335a8e420a1511e1969c8ff5518bebb748cf58b52e2559a47"} Oct 30 17:02:14.487871 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:14.487119 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" event={"ID":"c8682959-6953-4597-9c34-200732698947","Type":"ContainerStarted","Data":"837efdfdc8dc7f1b38914a9f78f524e3b43d9b14fcb059c384f701b1842cc98f"} Oct 30 17:02:14.574863 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:14.574813 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" podStartSLOduration=1.574798881 podStartE2EDuration="1.574798881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:14.572318413 +0000 UTC m=+2364.914992463" watchObservedRunningTime="2026-10-30 17:02:14.574798881 +0000 UTC m=+2364.917472979" Oct 30 17:02:15.015366 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:15.015333 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:15.017264 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:15.017235 4861 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 17:02:15.017406 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:15.017307 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" podUID="c8682959-6953-4597-9c34-200732698947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 17:02:16.024750 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:16.022153 4861 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 17:02:16.024750 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:16.022219 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" podUID="c8682959-6953-4597-9c34-200732698947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 17:02:17.017118 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:17.017080 4861 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 17:02:17.017428 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:17.017400 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" podUID="c8682959-6953-4597-9c34-200732698947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 17:02:18.020116 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:18.020074 4861 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 17:02:18.020755 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:18.020693 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" podUID="c8682959-6953-4597-9c34-200732698947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 17:02:19.017088 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:19.017052 4861 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 17:02:19.017499 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:19.017470 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" podUID="c8682959-6953-4597-9c34-200732698947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 17:02:20.025176 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:20.025131 4861 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body= Oct 30 17:02:20.025802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:20.025771 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" podUID="c8682959-6953-4597-9c34-200732698947" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" Oct 30 17:02:20.292343 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:02:20.292254 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:02:21.021738 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.021691 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:21.022203 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.022183 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:21.022975 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.022955 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:02:21.483225 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.483194 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv"] Oct 30 17:02:21.489246 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.489219 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.506658 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.506639 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-81733d\"/\"kube-root-ca.crt\"" Oct 30 17:02:21.507900 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.507878 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv"] Oct 30 17:02:21.518461 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.518422 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-81733d\"/\"openshift-service-ca.crt\"" Oct 30 17:02:21.518461 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.518450 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-81733d\"/\"default-dockercfg-s45qt\"" Oct 30 17:02:21.518632 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.518511 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-81733d\"/\"autores-model-81733d-kserve-self-signed-certs\"" Oct 30 17:02:21.595658 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.595629 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-home\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.595860 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.595669 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-tmp-dir\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.595860 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.595778 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-model-cache\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.595956 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.595893 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxkfk\" (UniqueName: \"kubernetes.io/projected/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-kube-api-access-xxkfk\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.595956 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.595934 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-tls-certs\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.595956 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.595952 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-dshm\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.696753 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.696706 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-tls-certs\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.696914 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.696762 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-dshm\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.696914 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.696907 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-home\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.697060 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.697019 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-tmp-dir\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.697156 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.697121 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-model-cache\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.697315 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.697284 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-home\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.697463 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.697292 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxkfk\" (UniqueName: \"kubernetes.io/projected/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-kube-api-access-xxkfk\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.697463 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.697354 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-tmp-dir\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.697571 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.697459 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-model-cache\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.699390 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.699368 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-dshm\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.699665 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.699648 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-tls-certs\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.723217 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.723194 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxkfk\" (UniqueName: \"kubernetes.io/projected/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-kube-api-access-xxkfk\") pod \"autores-model-81733d-kserve-d96b89849-pwqfv\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:21.817801 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:21.815185 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:22.145575 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:22.145538 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv"] Oct 30 17:02:22.146872 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:02:22.146845 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod196d83e3_a442_4d21_b20c_6d0fc9d6d65b.slice/crio-ce839ed343f01724100cd94f74a9b0276f9b1e54180cbd5242a7df45e2a8a80e WatchSource:0}: Error finding container ce839ed343f01724100cd94f74a9b0276f9b1e54180cbd5242a7df45e2a8a80e: Status 404 returned error can't find the container with id ce839ed343f01724100cd94f74a9b0276f9b1e54180cbd5242a7df45e2a8a80e Oct 30 17:02:22.569465 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:22.569426 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" event={"ID":"196d83e3-a442-4d21-b20c-6d0fc9d6d65b","Type":"ContainerStarted","Data":"00f6f6ede5595d16b320d8f7f3519b5d9408140a51f4d042a73ee16370eb7616"} Oct 30 17:02:22.570010 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:22.569475 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" event={"ID":"196d83e3-a442-4d21-b20c-6d0fc9d6d65b","Type":"ContainerStarted","Data":"ce839ed343f01724100cd94f74a9b0276f9b1e54180cbd5242a7df45e2a8a80e"} Oct 30 17:02:22.617540 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:22.617485 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" podStartSLOduration=1.617470157 podStartE2EDuration="1.617470157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:22.61589556 +0000 UTC m=+2372.958569600" watchObservedRunningTime="2026-10-30 17:02:22.617470157 +0000 UTC m=+2372.960144195" Oct 30 17:02:22.818373 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:22.818341 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:31.819182 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:31.819101 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:31.835346 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:31.835326 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:32.674054 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:32.674022 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:02:41.138404 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.138372 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr"] Oct 30 17:02:41.145467 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.145443 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:02:41.163913 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.163885 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr"] Oct 30 17:02:41.171949 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.171919 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-5758b7-serving-cert\"" Oct 30 17:02:41.196432 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.196403 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lvng\" (UniqueName: \"kubernetes.io/projected/276abb2e-f978-4d16-bfd4-c8838f173e31-kube-api-access-2lvng\") pod \"maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr\" (UID: \"276abb2e-f978-4d16-bfd4-c8838f173e31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:02:41.196635 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.196487 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/276abb2e-f978-4d16-bfd4-c8838f173e31-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr\" (UID: \"276abb2e-f978-4d16-bfd4-c8838f173e31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:02:41.196635 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.196556 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276abb2e-f978-4d16-bfd4-c8838f173e31-metrics-certs\") pod \"maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr\" (UID: \"276abb2e-f978-4d16-bfd4-c8838f173e31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:02:41.196828 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.196706 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276abb2e-f978-4d16-bfd4-c8838f173e31-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr\" (UID: \"276abb2e-f978-4d16-bfd4-c8838f173e31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:02:41.297851 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.297823 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lvng\" (UniqueName: \"kubernetes.io/projected/276abb2e-f978-4d16-bfd4-c8838f173e31-kube-api-access-2lvng\") pod \"maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr\" (UID: \"276abb2e-f978-4d16-bfd4-c8838f173e31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:02:41.298052 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.297901 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/276abb2e-f978-4d16-bfd4-c8838f173e31-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr\" (UID: \"276abb2e-f978-4d16-bfd4-c8838f173e31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:02:41.298052 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.298006 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276abb2e-f978-4d16-bfd4-c8838f173e31-metrics-certs\") pod \"maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr\" (UID: \"276abb2e-f978-4d16-bfd4-c8838f173e31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:02:41.298201 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.298058 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276abb2e-f978-4d16-bfd4-c8838f173e31-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr\" (UID: \"276abb2e-f978-4d16-bfd4-c8838f173e31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:02:41.298648 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.298621 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/276abb2e-f978-4d16-bfd4-c8838f173e31-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr\" (UID: \"276abb2e-f978-4d16-bfd4-c8838f173e31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:02:41.300788 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.300766 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276abb2e-f978-4d16-bfd4-c8838f173e31-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr\" (UID: \"276abb2e-f978-4d16-bfd4-c8838f173e31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:02:41.300901 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.300823 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276abb2e-f978-4d16-bfd4-c8838f173e31-metrics-certs\") pod \"maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr\" (UID: \"276abb2e-f978-4d16-bfd4-c8838f173e31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:02:41.321489 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.321465 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lvng\" (UniqueName: \"kubernetes.io/projected/276abb2e-f978-4d16-bfd4-c8838f173e31-kube-api-access-2lvng\") pod \"maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr\" (UID: \"276abb2e-f978-4d16-bfd4-c8838f173e31\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:02:41.475796 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.468024 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:02:41.875756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:41.871273 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr"] Oct 30 17:02:42.776114 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:42.776068 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" event={"ID":"276abb2e-f978-4d16-bfd4-c8838f173e31","Type":"ContainerStarted","Data":"8daeb9c38418430fc2aa949446321f086c2c9bb40741d213db60be2674fbc5bf"} Oct 30 17:02:42.776665 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:42.776176 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" event={"ID":"276abb2e-f978-4d16-bfd4-c8838f173e31","Type":"ContainerStarted","Data":"4876a5ddd0444f0027458f2659a0f88e30858f8e56a0d6f9e4850de1c8f5aac6"} Oct 30 17:02:42.776665 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:42.776583 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:02:42.828544 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:42.828487 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" podStartSLOduration=1.382931086 podStartE2EDuration="1.82847418s" totalImagesPullingTime="445.543094ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:42.825183752 +0000 UTC m=+2393.167857792" watchObservedRunningTime="2026-10-30 17:02:42.82847418 +0000 UTC m=+2393.171148218" Oct 30 17:02:43.823088 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.823054 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn"] Oct 30 17:02:43.828151 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.828127 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.839550 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.839522 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmqbl\" (UniqueName: \"kubernetes.io/projected/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-kube-api-access-cmqbl\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.839676 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.839565 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.839676 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.839635 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.839818 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.839708 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.839875 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.839802 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.839875 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.839869 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.839978 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.839958 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.840138 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.840117 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.840312 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.840289 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.854496 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.854472 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn"] Oct 30 17:02:43.942166 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.942136 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.942341 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.942199 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.942341 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.942248 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.942341 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.942331 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmqbl\" (UniqueName: \"kubernetes.io/projected/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-kube-api-access-cmqbl\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.942548 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.942349 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.942548 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.942365 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.942548 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.942437 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.942548 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.942466 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.943041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.942542 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.943041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.942633 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-credential-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.943041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.942764 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-workload-socket\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.943041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.942914 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-workload-certs\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.943041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.942935 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-data\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.943360 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.943194 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istiod-ca-cert\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.944950 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.944926 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-envoy\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.945413 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.945395 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-podinfo\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.966039 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.966018 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-token\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:43.966188 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:43.966169 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmqbl\" (UniqueName: \"kubernetes.io/projected/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-kube-api-access-cmqbl\") pod \"e2e-autores-81733d-openshift-default-688ffd6987-rn7sn\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:44.145796 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:44.145763 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:44.507123 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:44.504753 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn"] Oct 30 17:02:44.521682 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:02:44.521645 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0185f6c0_b86f_43ee_aab2_1c1d3728c7e7.slice/crio-f13019b114755d22d944a7f7ecbd919c54d25cc11945fcfbb0d575096e38ecd7 WatchSource:0}: Error finding container f13019b114755d22d944a7f7ecbd919c54d25cc11945fcfbb0d575096e38ecd7: Status 404 returned error can't find the container with id f13019b114755d22d944a7f7ecbd919c54d25cc11945fcfbb0d575096e38ecd7 Oct 30 17:02:44.525045 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:44.525003 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:02:44.525171 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:44.525079 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:02:44.525171 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:44.525108 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:02:44.793602 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:44.793492 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" event={"ID":"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7","Type":"ContainerStarted","Data":"781ea3197b9ec394596c212d5cf9dbfa31f7b04a98cfd2aadac157658ec0c650"} Oct 30 17:02:44.793602 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:44.793565 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" event={"ID":"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7","Type":"ContainerStarted","Data":"f13019b114755d22d944a7f7ecbd919c54d25cc11945fcfbb0d575096e38ecd7"} Oct 30 17:02:44.843936 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:44.843876 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" podStartSLOduration=1.843862289 podStartE2EDuration="1.843862289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:44.841846648 +0000 UTC m=+2395.184520688" watchObservedRunningTime="2026-10-30 17:02:44.843862289 +0000 UTC m=+2395.186536326" Oct 30 17:02:45.146165 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:45.146055 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:45.147427 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:45.147398 4861 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 17:02:45.147596 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:45.147451 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" podUID="0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 17:02:46.146100 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:46.146064 4861 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 17:02:46.146527 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:46.146118 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" podUID="0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 17:02:47.028433 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.028391 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw"] Oct 30 17:02:47.033376 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.033352 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:02:47.045947 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.045919 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw"] Oct 30 17:02:47.054534 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.053231 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"agc-cred-4bf9743d880e\"" Oct 30 17:02:47.054534 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.053471 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"payload-processing-external-model-plugins-e2e-extmod-b-13c495\"" Oct 30 17:02:47.054534 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.053477 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"payload-processing-external-model-e2e--be0deb1b-dockercfg-vxb7r\"" Oct 30 17:02:47.054534 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.053599 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"kube-root-ca.crt\"" Oct 30 17:02:47.061116 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.061097 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"routing-overlay\"" Oct 30 17:02:47.061220 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.061128 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"openshift-service-ca.crt\"" Oct 30 17:02:47.163747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.162793 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99lbc\" (UniqueName: \"kubernetes.io/projected/df60f910-7e4b-4d21-ad99-7631726cb9d9-kube-api-access-99lbc\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw\" (UID: \"df60f910-7e4b-4d21-ad99-7631726cb9d9\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:02:47.163747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.162971 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df60f910-7e4b-4d21-ad99-7631726cb9d9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw\" (UID: \"df60f910-7e4b-4d21-ad99-7631726cb9d9\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:02:47.163747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.163048 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df60f910-7e4b-4d21-ad99-7631726cb9d9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw\" (UID: \"df60f910-7e4b-4d21-ad99-7631726cb9d9\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:02:47.163747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.163081 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df60f910-7e4b-4d21-ad99-7631726cb9d9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw\" (UID: \"df60f910-7e4b-4d21-ad99-7631726cb9d9\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:02:47.173758 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.172292 4861 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 17:02:47.173758 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.172346 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" podUID="0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 17:02:47.264795 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.264746 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df60f910-7e4b-4d21-ad99-7631726cb9d9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw\" (UID: \"df60f910-7e4b-4d21-ad99-7631726cb9d9\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:02:47.265014 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.264808 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df60f910-7e4b-4d21-ad99-7631726cb9d9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw\" (UID: \"df60f910-7e4b-4d21-ad99-7631726cb9d9\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:02:47.265014 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.264842 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df60f910-7e4b-4d21-ad99-7631726cb9d9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw\" (UID: \"df60f910-7e4b-4d21-ad99-7631726cb9d9\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:02:47.265152 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.265062 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99lbc\" (UniqueName: \"kubernetes.io/projected/df60f910-7e4b-4d21-ad99-7631726cb9d9-kube-api-access-99lbc\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw\" (UID: \"df60f910-7e4b-4d21-ad99-7631726cb9d9\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:02:47.266862 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.266824 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df60f910-7e4b-4d21-ad99-7631726cb9d9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw\" (UID: \"df60f910-7e4b-4d21-ad99-7631726cb9d9\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:02:47.267303 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.267285 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df60f910-7e4b-4d21-ad99-7631726cb9d9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw\" (UID: \"df60f910-7e4b-4d21-ad99-7631726cb9d9\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:02:47.268847 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.268824 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df60f910-7e4b-4d21-ad99-7631726cb9d9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw\" (UID: \"df60f910-7e4b-4d21-ad99-7631726cb9d9\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:02:47.289368 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.289313 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99lbc\" (UniqueName: \"kubernetes.io/projected/df60f910-7e4b-4d21-ad99-7631726cb9d9-kube-api-access-99lbc\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw\" (UID: \"df60f910-7e4b-4d21-ad99-7631726cb9d9\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:02:47.350822 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.350791 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:02:47.679870 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.679845 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw"] Oct 30 17:02:47.680467 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:02:47.680443 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf60f910_7e4b_4d21_ad99_7631726cb9d9.slice/crio-8472730cdbd77f15d99da15ff606d45465c09d917e40041f2905fb3be589b875 WatchSource:0}: Error finding container 8472730cdbd77f15d99da15ff606d45465c09d917e40041f2905fb3be589b875: Status 404 returned error can't find the container with id 8472730cdbd77f15d99da15ff606d45465c09d917e40041f2905fb3be589b875 Oct 30 17:02:47.894888 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.894857 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" event={"ID":"df60f910-7e4b-4d21-ad99-7631726cb9d9","Type":"ContainerStarted","Data":"23d543bd1263f06a80c2aec436ec69cfb490a37d3e129636f4c13f5e9072a103"} Oct 30 17:02:47.895068 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.894908 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" event={"ID":"df60f910-7e4b-4d21-ad99-7631726cb9d9","Type":"ContainerStarted","Data":"8472730cdbd77f15d99da15ff606d45465c09d917e40041f2905fb3be589b875"} Oct 30 17:02:47.895068 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.895044 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:02:47.943853 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:47.943762 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" podStartSLOduration=1.943717073 podStartE2EDuration="1.943717073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:47.942895808 +0000 UTC m=+2398.285569847" watchObservedRunningTime="2026-10-30 17:02:47.943717073 +0000 UTC m=+2398.286391115" Oct 30 17:02:48.146136 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:48.146105 4861 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 17:02:48.146320 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:48.146160 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" podUID="0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 17:02:48.790472 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:48.790444 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:02:49.146130 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:49.146052 4861 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 17:02:49.146130 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:49.146107 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" podUID="0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 17:02:50.146067 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:50.146030 4861 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 17:02:50.146548 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:50.146094 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" podUID="0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 17:02:50.189692 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:50.189664 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 17:02:50.242056 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:50.242032 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 17:02:50.291116 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:02:50.291081 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:02:51.150984 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:51.150949 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:51.151477 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:51.151431 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:51.152032 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:51.152014 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:02:52.896391 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:52.896341 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv"] Oct 30 17:02:58.949951 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:02:58.949591 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:03:03.675411 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:03.675370 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r"] Oct 30 17:03:07.923789 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:07.923741 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" podUID="196d83e3-a442-4d21-b20c-6d0fc9d6d65b" containerName="main" containerID="cri-o://00f6f6ede5595d16b320d8f7f3519b5d9408140a51f4d042a73ee16370eb7616" gracePeriod=45 Oct 30 17:03:08.075370 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.075338 4861 generic.go:356] "Generic (PLEG): container finished" podID="196d83e3-a442-4d21-b20c-6d0fc9d6d65b" containerID="00f6f6ede5595d16b320d8f7f3519b5d9408140a51f4d042a73ee16370eb7616" exitCode=0 Oct 30 17:03:08.075550 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.075420 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" event={"ID":"196d83e3-a442-4d21-b20c-6d0fc9d6d65b","Type":"ContainerDied","Data":"00f6f6ede5595d16b320d8f7f3519b5d9408140a51f4d042a73ee16370eb7616"} Oct 30 17:03:08.214194 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.214170 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:03:08.321174 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.321138 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-kube-api-access-xxkfk\" (UniqueName: \"kubernetes.io/projected/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-kube-api-access-xxkfk\") pod \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " Oct 30 17:03:08.321356 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.321197 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-tmp-dir\") pod \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " Oct 30 17:03:08.321356 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.321306 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-dshm\") pod \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " Oct 30 17:03:08.321484 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.321411 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-tls-certs\" (UniqueName: \"kubernetes.io/secret/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-tls-certs\") pod \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " Oct 30 17:03:08.321484 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.321472 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-home\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-home\") pod \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " Oct 30 17:03:08.321604 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.321489 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-tmp-dir" pod "196d83e3-a442-4d21-b20c-6d0fc9d6d65b" (UID: "196d83e3-a442-4d21-b20c-6d0fc9d6d65b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:08.321604 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.321565 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-model-cache\") pod \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\" (UID: \"196d83e3-a442-4d21-b20c-6d0fc9d6d65b\") " Oct 30 17:03:08.322046 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.321927 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-model-cache" pod "196d83e3-a442-4d21-b20c-6d0fc9d6d65b" (UID: "196d83e3-a442-4d21-b20c-6d0fc9d6d65b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:08.322046 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.321932 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-home" pod "196d83e3-a442-4d21-b20c-6d0fc9d6d65b" (UID: "196d83e3-a442-4d21-b20c-6d0fc9d6d65b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:08.322466 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.322443 4861 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:08.322555 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.322471 4861 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:08.322555 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.322487 4861 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:08.323614 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.323589 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-kube-api-access-xxkfk" pod "196d83e3-a442-4d21-b20c-6d0fc9d6d65b" (UID: "196d83e3-a442-4d21-b20c-6d0fc9d6d65b"). InnerVolumeSpecName "kube-api-access-xxkfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:08.323924 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.323896 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-tls-certs" pod "196d83e3-a442-4d21-b20c-6d0fc9d6d65b" (UID: "196d83e3-a442-4d21-b20c-6d0fc9d6d65b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:03:08.324025 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.323936 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-dshm" pod "196d83e3-a442-4d21-b20c-6d0fc9d6d65b" (UID: "196d83e3-a442-4d21-b20c-6d0fc9d6d65b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:08.424272 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.424242 4861 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:08.424272 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.424270 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxkfk\" (UniqueName: \"kubernetes.io/projected/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-kube-api-access-xxkfk\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:08.424272 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.424280 4861 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/196d83e3-a442-4d21-b20c-6d0fc9d6d65b-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:08.676524 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.676485 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" podUID="93a745a6-d568-4614-a1cf-2c2f20f3c550" containerName="payload-processing" containerID="cri-o://033d9929c193287147d22283560d74def7dd04f1d8dd62162ea712bf395b8410" gracePeriod=25 Oct 30 17:03:08.967095 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:08.967071 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" Oct 30 17:03:09.095978 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.095949 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" Oct 30 17:03:09.096172 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.095979 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv" event={"ID":"196d83e3-a442-4d21-b20c-6d0fc9d6d65b","Type":"ContainerDied","Data":"ce839ed343f01724100cd94f74a9b0276f9b1e54180cbd5242a7df45e2a8a80e"} Oct 30 17:03:09.096172 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.096031 4861 scope.go:122] "RemoveContainer" containerID="00f6f6ede5595d16b320d8f7f3519b5d9408140a51f4d042a73ee16370eb7616" Oct 30 17:03:09.097734 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.097651 4861 generic.go:356] "Generic (PLEG): container finished" podID="93a745a6-d568-4614-a1cf-2c2f20f3c550" containerID="033d9929c193287147d22283560d74def7dd04f1d8dd62162ea712bf395b8410" exitCode=0 Oct 30 17:03:09.097734 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.097698 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" Oct 30 17:03:09.097990 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.097773 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" event={"ID":"93a745a6-d568-4614-a1cf-2c2f20f3c550","Type":"ContainerDied","Data":"033d9929c193287147d22283560d74def7dd04f1d8dd62162ea712bf395b8410"} Oct 30 17:03:09.097990 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.097864 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r" event={"ID":"93a745a6-d568-4614-a1cf-2c2f20f3c550","Type":"ContainerDied","Data":"24ac873972e19599ddcb2515a70b36a7253a8ff484056a17e68e3733fecfe3f7"} Oct 30 17:03:09.108042 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.108022 4861 scope.go:122] "RemoveContainer" containerID="033d9929c193287147d22283560d74def7dd04f1d8dd62162ea712bf395b8410" Oct 30 17:03:09.119637 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.119615 4861 scope.go:122] "RemoveContainer" containerID="033d9929c193287147d22283560d74def7dd04f1d8dd62162ea712bf395b8410" Oct 30 17:03:09.120069 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:03:09.120045 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033d9929c193287147d22283560d74def7dd04f1d8dd62162ea712bf395b8410\": container with ID starting with 033d9929c193287147d22283560d74def7dd04f1d8dd62162ea712bf395b8410 not found: ID does not exist" containerID="033d9929c193287147d22283560d74def7dd04f1d8dd62162ea712bf395b8410" Oct 30 17:03:09.120171 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.120076 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033d9929c193287147d22283560d74def7dd04f1d8dd62162ea712bf395b8410"} err="failed to get container status \"033d9929c193287147d22283560d74def7dd04f1d8dd62162ea712bf395b8410\": rpc error: code = NotFound desc = could not find container \"033d9929c193287147d22283560d74def7dd04f1d8dd62162ea712bf395b8410\": container with ID starting with 033d9929c193287147d22283560d74def7dd04f1d8dd62162ea712bf395b8410 not found: ID does not exist" Oct 30 17:03:09.133311 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.133287 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a745a6-d568-4614-a1cf-2c2f20f3c550-kube-api-access-wlltt\" (UniqueName: \"kubernetes.io/projected/93a745a6-d568-4614-a1cf-2c2f20f3c550-kube-api-access-wlltt\") pod \"93a745a6-d568-4614-a1cf-2c2f20f3c550\" (UID: \"93a745a6-d568-4614-a1cf-2c2f20f3c550\") " Oct 30 17:03:09.133536 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.133522 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a745a6-d568-4614-a1cf-2c2f20f3c550-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a745a6-d568-4614-a1cf-2c2f20f3c550-plugins-config-volume\") pod \"93a745a6-d568-4614-a1cf-2c2f20f3c550\" (UID: \"93a745a6-d568-4614-a1cf-2c2f20f3c550\") " Oct 30 17:03:09.133915 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.133893 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a745a6-d568-4614-a1cf-2c2f20f3c550-plugins-config-volume" pod "93a745a6-d568-4614-a1cf-2c2f20f3c550" (UID: "93a745a6-d568-4614-a1cf-2c2f20f3c550"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:03:09.135546 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.135524 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a745a6-d568-4614-a1cf-2c2f20f3c550-kube-api-access-wlltt" pod "93a745a6-d568-4614-a1cf-2c2f20f3c550" (UID: "93a745a6-d568-4614-a1cf-2c2f20f3c550"). InnerVolumeSpecName "kube-api-access-wlltt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:09.151489 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.151466 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv"] Oct 30 17:03:09.154906 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.154887 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-81733d/autores-model-81733d-kserve-d96b89849-pwqfv"] Oct 30 17:03:09.235349 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.235310 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a745a6-d568-4614-a1cf-2c2f20f3c550-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:09.235349 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.235352 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlltt\" (UniqueName: \"kubernetes.io/projected/93a745a6-d568-4614-a1cf-2c2f20f3c550-kube-api-access-wlltt\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:09.445816 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.445772 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r"] Oct 30 17:03:09.453233 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.453204 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-81733d-859f6d68bd-p2f9r"] Oct 30 17:03:09.809155 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.809121 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="196d83e3-a442-4d21-b20c-6d0fc9d6d65b" path="/var/lib/kubelet/pods/196d83e3-a442-4d21-b20c-6d0fc9d6d65b/volumes" Oct 30 17:03:09.809680 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:09.809658 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a745a6-d568-4614-a1cf-2c2f20f3c550" path="/var/lib/kubelet/pods/93a745a6-d568-4614-a1cf-2c2f20f3c550/volumes" Oct 30 17:03:20.082144 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.082110 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq"] Oct 30 17:03:20.082921 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.082907 4861 cpu_manager.go:414] "containerMap: removing container" podUID="196d83e3-a442-4d21-b20c-6d0fc9d6d65b" containerName="main" Oct 30 17:03:20.082981 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.082922 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="196d83e3-a442-4d21-b20c-6d0fc9d6d65b" containerName="main" Oct 30 17:03:20.083025 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.082980 4861 cpu_manager.go:414] "containerMap: removing container" podUID="93a745a6-d568-4614-a1cf-2c2f20f3c550" containerName="payload-processing" Oct 30 17:03:20.083025 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.082985 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a745a6-d568-4614-a1cf-2c2f20f3c550" containerName="payload-processing" Oct 30 17:03:20.083112 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.083102 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a745a6-d568-4614-a1cf-2c2f20f3c550" containerName="payload-processing" Oct 30 17:03:20.083149 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.083112 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="196d83e3-a442-4d21-b20c-6d0fc9d6d65b" containerName="main" Oct 30 17:03:20.087038 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.087020 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:03:20.107615 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.107595 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq"] Oct 30 17:03:20.118945 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.117007 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"routing-overlay\"" Oct 30 17:03:20.118945 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.117079 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"payload-processing-external-model-e2e--22b3ecb4-dockercfg-6tsff\"" Oct 30 17:03:20.118945 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.117125 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"kube-root-ca.crt\"" Oct 30 17:03:20.118945 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.117187 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5c6b3c\"" Oct 30 17:03:20.118945 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.117262 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"openshift-service-ca.crt\"" Oct 30 17:03:20.118945 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.117331 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"agc-cred-3e1593902499\"" Oct 30 17:03:20.151670 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.151636 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35fc2733-2e52-469c-a59f-c7eae395d4cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq\" (UID: \"35fc2733-2e52-469c-a59f-c7eae395d4cd\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:03:20.152026 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.151994 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mc55\" (UniqueName: \"kubernetes.io/projected/35fc2733-2e52-469c-a59f-c7eae395d4cd-kube-api-access-2mc55\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq\" (UID: \"35fc2733-2e52-469c-a59f-c7eae395d4cd\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:03:20.152134 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.152060 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35fc2733-2e52-469c-a59f-c7eae395d4cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq\" (UID: \"35fc2733-2e52-469c-a59f-c7eae395d4cd\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:03:20.152134 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.152119 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35fc2733-2e52-469c-a59f-c7eae395d4cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq\" (UID: \"35fc2733-2e52-469c-a59f-c7eae395d4cd\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:03:20.261658 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.261608 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35fc2733-2e52-469c-a59f-c7eae395d4cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq\" (UID: \"35fc2733-2e52-469c-a59f-c7eae395d4cd\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:03:20.261914 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.261695 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35fc2733-2e52-469c-a59f-c7eae395d4cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq\" (UID: \"35fc2733-2e52-469c-a59f-c7eae395d4cd\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:03:20.262155 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.261999 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35fc2733-2e52-469c-a59f-c7eae395d4cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq\" (UID: \"35fc2733-2e52-469c-a59f-c7eae395d4cd\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:03:20.262282 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.262201 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mc55\" (UniqueName: \"kubernetes.io/projected/35fc2733-2e52-469c-a59f-c7eae395d4cd-kube-api-access-2mc55\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq\" (UID: \"35fc2733-2e52-469c-a59f-c7eae395d4cd\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:03:20.263601 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.263574 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35fc2733-2e52-469c-a59f-c7eae395d4cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq\" (UID: \"35fc2733-2e52-469c-a59f-c7eae395d4cd\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:03:20.265133 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.264380 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35fc2733-2e52-469c-a59f-c7eae395d4cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq\" (UID: \"35fc2733-2e52-469c-a59f-c7eae395d4cd\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:03:20.270499 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.270473 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35fc2733-2e52-469c-a59f-c7eae395d4cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq\" (UID: \"35fc2733-2e52-469c-a59f-c7eae395d4cd\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:03:20.288466 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.288437 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mc55\" (UniqueName: \"kubernetes.io/projected/35fc2733-2e52-469c-a59f-c7eae395d4cd-kube-api-access-2mc55\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq\" (UID: \"35fc2733-2e52-469c-a59f-c7eae395d4cd\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:03:20.310169 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:03:20.310137 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:03:20.402620 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.402550 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:03:20.713853 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:20.713824 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq"] Oct 30 17:03:20.715018 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:03:20.714968 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fc2733_2e52_469c_a59f_c7eae395d4cd.slice/crio-cee8957f629a07363760cb4fb968888b870df984ab69a6e2258ec17373420ffb WatchSource:0}: Error finding container cee8957f629a07363760cb4fb968888b870df984ab69a6e2258ec17373420ffb: Status 404 returned error can't find the container with id cee8957f629a07363760cb4fb968888b870df984ab69a6e2258ec17373420ffb Oct 30 17:03:21.233410 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:21.233374 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" event={"ID":"35fc2733-2e52-469c-a59f-c7eae395d4cd","Type":"ContainerStarted","Data":"a64ef0fa892f9991062c3f0c7231a9b90bd9dfb3380dda3aff3c62f578dc085e"} Oct 30 17:03:21.233862 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:21.233441 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:03:21.233862 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:21.233469 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" event={"ID":"35fc2733-2e52-469c-a59f-c7eae395d4cd","Type":"ContainerStarted","Data":"cee8957f629a07363760cb4fb968888b870df984ab69a6e2258ec17373420ffb"} Oct 30 17:03:21.277360 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:21.277313 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" podStartSLOduration=1.277299582 podStartE2EDuration="1.277299582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:21.276239528 +0000 UTC m=+2431.618913568" watchObservedRunningTime="2026-10-30 17:03:21.277299582 +0000 UTC m=+2431.619973619" Oct 30 17:03:31.601137 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.600992 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6"] Oct 30 17:03:31.606609 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.606584 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:03:31.622629 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.622606 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"kube-root-ca.crt\"" Oct 30 17:03:31.623153 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.623130 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6"] Oct 30 17:03:31.630963 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.630859 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"routing-overlay\"" Oct 30 17:03:31.630963 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.630913 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5758b7\"" Oct 30 17:03:31.631156 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.630974 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"openshift-service-ca.crt\"" Oct 30 17:03:31.639061 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.639041 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"agc-cred-9d6cc5c0b99e\"" Oct 30 17:03:31.657031 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.657005 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1536eff9-3744-4068-ae94-f2063461b26f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6\" (UID: \"1536eff9-3744-4068-ae94-f2063461b26f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:03:31.657148 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.657053 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl5pv\" (UniqueName: \"kubernetes.io/projected/1536eff9-3744-4068-ae94-f2063461b26f-kube-api-access-hl5pv\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6\" (UID: \"1536eff9-3744-4068-ae94-f2063461b26f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:03:31.657148 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.657094 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1536eff9-3744-4068-ae94-f2063461b26f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6\" (UID: \"1536eff9-3744-4068-ae94-f2063461b26f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:03:31.657239 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.657211 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1536eff9-3744-4068-ae94-f2063461b26f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6\" (UID: \"1536eff9-3744-4068-ae94-f2063461b26f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:03:31.758272 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.758246 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1536eff9-3744-4068-ae94-f2063461b26f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6\" (UID: \"1536eff9-3744-4068-ae94-f2063461b26f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:03:31.758459 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.758287 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl5pv\" (UniqueName: \"kubernetes.io/projected/1536eff9-3744-4068-ae94-f2063461b26f-kube-api-access-hl5pv\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6\" (UID: \"1536eff9-3744-4068-ae94-f2063461b26f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:03:31.758459 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.758316 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1536eff9-3744-4068-ae94-f2063461b26f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6\" (UID: \"1536eff9-3744-4068-ae94-f2063461b26f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:03:31.758610 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.758469 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1536eff9-3744-4068-ae94-f2063461b26f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6\" (UID: \"1536eff9-3744-4068-ae94-f2063461b26f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:03:31.759176 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.759148 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1536eff9-3744-4068-ae94-f2063461b26f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6\" (UID: \"1536eff9-3744-4068-ae94-f2063461b26f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:03:31.759176 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.759165 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1536eff9-3744-4068-ae94-f2063461b26f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6\" (UID: \"1536eff9-3744-4068-ae94-f2063461b26f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:03:31.760970 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.760953 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1536eff9-3744-4068-ae94-f2063461b26f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6\" (UID: \"1536eff9-3744-4068-ae94-f2063461b26f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:03:31.782419 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.782389 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl5pv\" (UniqueName: \"kubernetes.io/projected/1536eff9-3744-4068-ae94-f2063461b26f-kube-api-access-hl5pv\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6\" (UID: \"1536eff9-3744-4068-ae94-f2063461b26f\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:03:31.926158 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:31.926067 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:03:32.240553 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:32.240524 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6"] Oct 30 17:03:32.241605 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:03:32.241581 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1536eff9_3744_4068_ae94_f2063461b26f.slice/crio-2fe79ac47113f7cc80cfbcd6a2e53f1684b57caf5aa8d787e7f0aa0ecece57a3 WatchSource:0}: Error finding container 2fe79ac47113f7cc80cfbcd6a2e53f1684b57caf5aa8d787e7f0aa0ecece57a3: Status 404 returned error can't find the container with id 2fe79ac47113f7cc80cfbcd6a2e53f1684b57caf5aa8d787e7f0aa0ecece57a3 Oct 30 17:03:32.244576 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:32.244556 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:03:32.355980 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:32.355946 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" event={"ID":"1536eff9-3744-4068-ae94-f2063461b26f","Type":"ContainerStarted","Data":"d24954d430504ca1c05c8311ccfa363f4a2eaf67b9417b2db065f3ef812e2c0c"} Oct 30 17:03:32.356157 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:32.356009 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:03:32.356157 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:32.356030 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" event={"ID":"1536eff9-3744-4068-ae94-f2063461b26f","Type":"ContainerStarted","Data":"2fe79ac47113f7cc80cfbcd6a2e53f1684b57caf5aa8d787e7f0aa0ecece57a3"} Oct 30 17:03:32.419453 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:32.419405 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" podStartSLOduration=1.419389818 podStartE2EDuration="1.419389818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:32.416993411 +0000 UTC m=+2442.759667450" watchObservedRunningTime="2026-10-30 17:03:32.419389818 +0000 UTC m=+2442.762064131" Oct 30 17:03:42.415828 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:42.415796 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq"] Oct 30 17:03:42.416389 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:42.415995 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" podUID="0697e319-8f24-4056-a1a7-e9782794dd7f" containerName="maas-api" containerID="cri-o://72b323ca724be6344e2310742aa6da25140949c6469cb9498fdeb48a1b5f2dc8" gracePeriod=30 Oct 30 17:03:42.775266 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:42.775235 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:03:42.951478 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:42.951440 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0697e319-8f24-4056-a1a7-e9782794dd7f-kube-api-access-dgq5w\" (UniqueName: \"kubernetes.io/projected/0697e319-8f24-4056-a1a7-e9782794dd7f-kube-api-access-dgq5w\") pod \"0697e319-8f24-4056-a1a7-e9782794dd7f\" (UID: \"0697e319-8f24-4056-a1a7-e9782794dd7f\") " Oct 30 17:03:42.951702 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:42.951494 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0697e319-8f24-4056-a1a7-e9782794dd7f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0697e319-8f24-4056-a1a7-e9782794dd7f-maas-api-tls\") pod \"0697e319-8f24-4056-a1a7-e9782794dd7f\" (UID: \"0697e319-8f24-4056-a1a7-e9782794dd7f\") " Oct 30 17:03:42.951819 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:42.951710 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0697e319-8f24-4056-a1a7-e9782794dd7f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0697e319-8f24-4056-a1a7-e9782794dd7f-openshift-service-ca\") pod \"0697e319-8f24-4056-a1a7-e9782794dd7f\" (UID: \"0697e319-8f24-4056-a1a7-e9782794dd7f\") " Oct 30 17:03:42.951819 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:42.951810 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0697e319-8f24-4056-a1a7-e9782794dd7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0697e319-8f24-4056-a1a7-e9782794dd7f-metrics-certs\") pod \"0697e319-8f24-4056-a1a7-e9782794dd7f\" (UID: \"0697e319-8f24-4056-a1a7-e9782794dd7f\") " Oct 30 17:03:42.971975 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:42.971924 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0697e319-8f24-4056-a1a7-e9782794dd7f-kube-api-access-dgq5w" pod "0697e319-8f24-4056-a1a7-e9782794dd7f" (UID: "0697e319-8f24-4056-a1a7-e9782794dd7f"). InnerVolumeSpecName "kube-api-access-dgq5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:42.978942 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:42.978694 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0697e319-8f24-4056-a1a7-e9782794dd7f-openshift-service-ca" pod "0697e319-8f24-4056-a1a7-e9782794dd7f" (UID: "0697e319-8f24-4056-a1a7-e9782794dd7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:03:42.981574 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:42.981531 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0697e319-8f24-4056-a1a7-e9782794dd7f-metrics-certs" pod "0697e319-8f24-4056-a1a7-e9782794dd7f" (UID: "0697e319-8f24-4056-a1a7-e9782794dd7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:03:42.982414 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:42.982376 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0697e319-8f24-4056-a1a7-e9782794dd7f-maas-api-tls" pod "0697e319-8f24-4056-a1a7-e9782794dd7f" (UID: "0697e319-8f24-4056-a1a7-e9782794dd7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:03:43.054457 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.054372 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgq5w\" (UniqueName: \"kubernetes.io/projected/0697e319-8f24-4056-a1a7-e9782794dd7f-kube-api-access-dgq5w\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:43.054457 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.054402 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0697e319-8f24-4056-a1a7-e9782794dd7f-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:43.054457 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.054411 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0697e319-8f24-4056-a1a7-e9782794dd7f-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:43.054457 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.054419 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0697e319-8f24-4056-a1a7-e9782794dd7f-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:43.336589 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.336515 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85"] Oct 30 17:03:43.337290 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.337275 4861 cpu_manager.go:414] "containerMap: removing container" podUID="0697e319-8f24-4056-a1a7-e9782794dd7f" containerName="maas-api" Oct 30 17:03:43.337351 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.337291 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0697e319-8f24-4056-a1a7-e9782794dd7f" containerName="maas-api" Oct 30 17:03:43.337476 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.337466 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="0697e319-8f24-4056-a1a7-e9782794dd7f" containerName="maas-api" Oct 30 17:03:43.341494 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.341473 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:03:43.360800 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.360777 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85"] Oct 30 17:03:43.367696 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.367676 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:03:43.369564 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.369536 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"openshift-service-ca.crt\"" Oct 30 17:03:43.369564 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.369550 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"agc-cred-61b8dd482a81\"" Oct 30 17:03:43.369781 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.369642 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"payload-processing-external-model-e2e--dc82f608-dockercfg-sdnt8\"" Oct 30 17:03:43.369781 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.369657 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"payload-processing-external-model-plugins-e2e-extmod-b-abda73\"" Oct 30 17:03:43.369781 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.369761 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"kube-root-ca.crt\"" Oct 30 17:03:43.369781 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.369762 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"agc-cred-7f008a3cd8f8\"" Oct 30 17:03:43.370028 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.369862 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-abda73\"/\"routing-overlay\"" Oct 30 17:03:43.445453 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.445413 4861 generic.go:356] "Generic (PLEG): container finished" podID="0697e319-8f24-4056-a1a7-e9782794dd7f" containerID="72b323ca724be6344e2310742aa6da25140949c6469cb9498fdeb48a1b5f2dc8" exitCode=0 Oct 30 17:03:43.445980 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.445479 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" Oct 30 17:03:43.445980 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.445486 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" event={"ID":"0697e319-8f24-4056-a1a7-e9782794dd7f","Type":"ContainerDied","Data":"72b323ca724be6344e2310742aa6da25140949c6469cb9498fdeb48a1b5f2dc8"} Oct 30 17:03:43.445980 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.445553 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq" event={"ID":"0697e319-8f24-4056-a1a7-e9782794dd7f","Type":"ContainerDied","Data":"4e3fc37819f43e5d34bbe591e31c9c073e2a7a1ec38eb3f3c5c58b6304072972"} Oct 30 17:03:43.445980 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.445588 4861 scope.go:122] "RemoveContainer" containerID="72b323ca724be6344e2310742aa6da25140949c6469cb9498fdeb48a1b5f2dc8" Oct 30 17:03:43.461450 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.461314 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb605e3-c317-4b71-9262-9a27d0868d78-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85\" (UID: \"fbb605e3-c317-4b71-9262-9a27d0868d78\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:03:43.461591 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.461429 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbb605e3-c317-4b71-9262-9a27d0868d78-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85\" (UID: \"fbb605e3-c317-4b71-9262-9a27d0868d78\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:03:43.462767 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.461763 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6lnz\" (UniqueName: \"kubernetes.io/projected/fbb605e3-c317-4b71-9262-9a27d0868d78-kube-api-access-m6lnz\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85\" (UID: \"fbb605e3-c317-4b71-9262-9a27d0868d78\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:03:43.462767 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.461938 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbb605e3-c317-4b71-9262-9a27d0868d78-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85\" (UID: \"fbb605e3-c317-4b71-9262-9a27d0868d78\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:03:43.463291 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.463275 4861 scope.go:122] "RemoveContainer" containerID="72b323ca724be6344e2310742aa6da25140949c6469cb9498fdeb48a1b5f2dc8" Oct 30 17:03:43.463678 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:03:43.463654 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b323ca724be6344e2310742aa6da25140949c6469cb9498fdeb48a1b5f2dc8\": container with ID starting with 72b323ca724be6344e2310742aa6da25140949c6469cb9498fdeb48a1b5f2dc8 not found: ID does not exist" containerID="72b323ca724be6344e2310742aa6da25140949c6469cb9498fdeb48a1b5f2dc8" Oct 30 17:03:43.463804 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.463685 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b323ca724be6344e2310742aa6da25140949c6469cb9498fdeb48a1b5f2dc8"} err="failed to get container status \"72b323ca724be6344e2310742aa6da25140949c6469cb9498fdeb48a1b5f2dc8\": rpc error: code = NotFound desc = could not find container \"72b323ca724be6344e2310742aa6da25140949c6469cb9498fdeb48a1b5f2dc8\": container with ID starting with 72b323ca724be6344e2310742aa6da25140949c6469cb9498fdeb48a1b5f2dc8 not found: ID does not exist" Oct 30 17:03:43.514344 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.514307 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq"] Oct 30 17:03:43.518103 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.518078 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-81733d-5ccfdddd5d-ssttq"] Oct 30 17:03:43.563604 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.563572 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb605e3-c317-4b71-9262-9a27d0868d78-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85\" (UID: \"fbb605e3-c317-4b71-9262-9a27d0868d78\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:03:43.563808 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.563641 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbb605e3-c317-4b71-9262-9a27d0868d78-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85\" (UID: \"fbb605e3-c317-4b71-9262-9a27d0868d78\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:03:43.563911 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.563817 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6lnz\" (UniqueName: \"kubernetes.io/projected/fbb605e3-c317-4b71-9262-9a27d0868d78-kube-api-access-m6lnz\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85\" (UID: \"fbb605e3-c317-4b71-9262-9a27d0868d78\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:03:43.563988 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.563944 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbb605e3-c317-4b71-9262-9a27d0868d78-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85\" (UID: \"fbb605e3-c317-4b71-9262-9a27d0868d78\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:03:43.564385 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.564361 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb605e3-c317-4b71-9262-9a27d0868d78-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85\" (UID: \"fbb605e3-c317-4b71-9262-9a27d0868d78\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:03:43.564528 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.564508 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbb605e3-c317-4b71-9262-9a27d0868d78-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85\" (UID: \"fbb605e3-c317-4b71-9262-9a27d0868d78\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:03:43.566353 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.566331 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbb605e3-c317-4b71-9262-9a27d0868d78-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85\" (UID: \"fbb605e3-c317-4b71-9262-9a27d0868d78\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:03:43.609599 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.609517 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6lnz\" (UniqueName: \"kubernetes.io/projected/fbb605e3-c317-4b71-9262-9a27d0868d78-kube-api-access-m6lnz\") pod \"payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85\" (UID: \"fbb605e3-c317-4b71-9262-9a27d0868d78\") " pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:03:43.657634 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.657606 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:03:43.809191 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.809155 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0697e319-8f24-4056-a1a7-e9782794dd7f" path="/var/lib/kubelet/pods/0697e319-8f24-4056-a1a7-e9782794dd7f/volumes" Oct 30 17:03:43.968211 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:43.968188 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85"] Oct 30 17:03:43.968864 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:03:43.968838 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb605e3_c317_4b71_9262_9a27d0868d78.slice/crio-2714646188f861f51478241b9e86bd98f83e5756dc6412b065869ded15ac0ef9 WatchSource:0}: Error finding container 2714646188f861f51478241b9e86bd98f83e5756dc6412b065869ded15ac0ef9: Status 404 returned error can't find the container with id 2714646188f861f51478241b9e86bd98f83e5756dc6412b065869ded15ac0ef9 Oct 30 17:03:44.462154 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:44.462116 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" event={"ID":"fbb605e3-c317-4b71-9262-9a27d0868d78","Type":"ContainerStarted","Data":"6c16d5dea18fcb29bb071d8dbe395df0aa272580e6a013b4dbcce2d9e5e5c993"} Oct 30 17:03:44.462651 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:44.462180 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" event={"ID":"fbb605e3-c317-4b71-9262-9a27d0868d78","Type":"ContainerStarted","Data":"2714646188f861f51478241b9e86bd98f83e5756dc6412b065869ded15ac0ef9"} Oct 30 17:03:44.462651 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:44.462262 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:03:44.507893 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:44.507845 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" podStartSLOduration=1.507831998 podStartE2EDuration="1.507831998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:44.506964577 +0000 UTC m=+2454.849638616" watchObservedRunningTime="2026-10-30 17:03:44.507831998 +0000 UTC m=+2454.850506036" Oct 30 17:03:50.292619 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:03:50.292580 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:03:50.833773 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:50.833697 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq"] Oct 30 17:03:50.834994 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:50.834958 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" podUID="c8682959-6953-4597-9c34-200732698947" containerName="istio-proxy" containerID="cri-o://c8ab7e8f793235e335a8e420a1511e1969c8ff5518bebb748cf58b52e2559a47" gracePeriod=30 Oct 30 17:03:50.853579 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:50.853544 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn"] Oct 30 17:03:50.853878 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:50.853852 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" podUID="0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" containerName="istio-proxy" containerID="cri-o://781ea3197b9ec394596c212d5cf9dbfa31f7b04a98cfd2aadac157658ec0c650" gracePeriod=30 Oct 30 17:03:51.025354 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:51.025312 4861 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:03:51.025533 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:51.025396 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" podUID="c8682959-6953-4597-9c34-200732698947" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:03:51.153594 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:51.153516 4861 patch_prober.go:29] interesting pod/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:03:51.153594 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:51.153578 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" podUID="0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:03:55.473912 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:55.473884 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:03:56.209516 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.209485 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:03:56.233303 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.233276 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:03:56.271355 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.271326 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8682959-6953-4597-9c34-200732698947-kube-api-access-tzjxc\" (UniqueName: \"kubernetes.io/projected/c8682959-6953-4597-9c34-200732698947-kube-api-access-tzjxc\") pod \"c8682959-6953-4597-9c34-200732698947\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " Oct 30 17:03:56.271637 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.271616 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-workload-socket\") pod \"c8682959-6953-4597-9c34-200732698947\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " Oct 30 17:03:56.271845 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.271825 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-workload-certs\") pod \"c8682959-6953-4597-9c34-200732698947\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " Oct 30 17:03:56.272026 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.272002 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8682959-6953-4597-9c34-200732698947-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8682959-6953-4597-9c34-200732698947-istiod-ca-cert\") pod \"c8682959-6953-4597-9c34-200732698947\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " Oct 30 17:03:56.272138 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.272004 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-workload-socket" pod "c8682959-6953-4597-9c34-200732698947" (UID: "c8682959-6953-4597-9c34-200732698947"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.272252 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.272233 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-workload-certs" pod "c8682959-6953-4597-9c34-200732698947" (UID: "c8682959-6953-4597-9c34-200732698947"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.273460 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.272518 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8682959-6953-4597-9c34-200732698947-istiod-ca-cert" pod "c8682959-6953-4597-9c34-200732698947" (UID: "c8682959-6953-4597-9c34-200732698947"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:03:56.273460 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.272641 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-credential-socket\") pod \"c8682959-6953-4597-9c34-200732698947\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " Oct 30 17:03:56.273460 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.272699 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-istio-envoy\") pod \"c8682959-6953-4597-9c34-200732698947\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " Oct 30 17:03:56.273460 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.273145 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8682959-6953-4597-9c34-200732698947-istio-token\" (UniqueName: \"kubernetes.io/projected/c8682959-6953-4597-9c34-200732698947-istio-token\") pod \"c8682959-6953-4597-9c34-200732698947\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " Oct 30 17:03:56.273460 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.273204 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-istio-data\") pod \"c8682959-6953-4597-9c34-200732698947\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " Oct 30 17:03:56.273460 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.273255 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8682959-6953-4597-9c34-200732698947-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8682959-6953-4597-9c34-200732698947-istio-podinfo\") pod \"c8682959-6953-4597-9c34-200732698947\" (UID: \"c8682959-6953-4597-9c34-200732698947\") " Oct 30 17:03:56.273991 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.273554 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-istio-data" pod "c8682959-6953-4597-9c34-200732698947" (UID: "c8682959-6953-4597-9c34-200732698947"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.273991 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.273967 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.273991 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.273987 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.274180 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.273999 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.274180 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.274011 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8682959-6953-4597-9c34-200732698947-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.275018 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.274985 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8682959-6953-4597-9c34-200732698947-kube-api-access-tzjxc" pod "c8682959-6953-4597-9c34-200732698947" (UID: "c8682959-6953-4597-9c34-200732698947"). InnerVolumeSpecName "kube-api-access-tzjxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:56.275313 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.275211 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-credential-socket" pod "c8682959-6953-4597-9c34-200732698947" (UID: "c8682959-6953-4597-9c34-200732698947"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.276666 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.276627 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8682959-6953-4597-9c34-200732698947-istio-token" pod "c8682959-6953-4597-9c34-200732698947" (UID: "c8682959-6953-4597-9c34-200732698947"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:56.277523 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.277493 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-istio-envoy" pod "c8682959-6953-4597-9c34-200732698947" (UID: "c8682959-6953-4597-9c34-200732698947"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.277856 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.277832 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8682959-6953-4597-9c34-200732698947-istio-podinfo" pod "c8682959-6953-4597-9c34-200732698947" (UID: "c8682959-6953-4597-9c34-200732698947"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:03:56.375364 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.375331 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-kube-api-access-cmqbl\" (UniqueName: \"kubernetes.io/projected/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-kube-api-access-cmqbl\") pod \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " Oct 30 17:03:56.375536 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.375412 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-credential-socket\") pod \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " Oct 30 17:03:56.375536 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.375520 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-podinfo\") pod \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " Oct 30 17:03:56.375629 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.375555 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istiod-ca-cert\") pod \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " Oct 30 17:03:56.375629 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.375583 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-workload-certs\") pod \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " Oct 30 17:03:56.375781 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.375649 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-token\" (UniqueName: \"kubernetes.io/projected/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-token\") pod \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " Oct 30 17:03:56.375781 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.375688 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-data\") pod \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " Oct 30 17:03:56.375781 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.375772 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-credential-socket" pod "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" (UID: "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.375969 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.375819 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-envoy\") pod \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " Oct 30 17:03:56.375969 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.375888 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-workload-socket\") pod \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\" (UID: \"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7\") " Oct 30 17:03:56.376085 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.376023 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-workload-certs" pod "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" (UID: "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.376153 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.376086 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istiod-ca-cert" pod "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" (UID: "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:03:56.376409 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.376369 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-workload-socket" pod "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" (UID: "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.376409 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.376413 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-data" pod "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" (UID: "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.376769 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.376696 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.376769 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.376752 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.376769 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.376767 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.377047 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.376781 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8682959-6953-4597-9c34-200732698947-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.377047 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.376794 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.377047 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.376805 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.377047 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.376820 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8682959-6953-4597-9c34-200732698947-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.377047 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.376834 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8682959-6953-4597-9c34-200732698947-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.377047 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.376846 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.377047 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.376862 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzjxc\" (UniqueName: \"kubernetes.io/projected/c8682959-6953-4597-9c34-200732698947-kube-api-access-tzjxc\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.378540 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.378519 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-envoy" pod "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" (UID: "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:56.378755 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.378708 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-podinfo" pod "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" (UID: "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:03:56.378820 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.378805 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-kube-api-access-cmqbl" pod "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" (UID: "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7"). InnerVolumeSpecName "kube-api-access-cmqbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:56.378859 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.378837 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-token" pod "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" (UID: "0185f6c0-b86f-43ee-aab2-1c1d3728c7e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:56.478377 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.478336 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.478377 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.478370 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.478377 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.478382 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.478875 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.478394 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmqbl\" (UniqueName: \"kubernetes.io/projected/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7-kube-api-access-cmqbl\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:03:56.594894 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.594863 4861 generic.go:356] "Generic (PLEG): container finished" podID="c8682959-6953-4597-9c34-200732698947" containerID="c8ab7e8f793235e335a8e420a1511e1969c8ff5518bebb748cf58b52e2559a47" exitCode=0 Oct 30 17:03:56.595099 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.594924 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" Oct 30 17:03:56.595099 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.594948 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" event={"ID":"c8682959-6953-4597-9c34-200732698947","Type":"ContainerDied","Data":"c8ab7e8f793235e335a8e420a1511e1969c8ff5518bebb748cf58b52e2559a47"} Oct 30 17:03:56.595099 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.595023 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq" event={"ID":"c8682959-6953-4597-9c34-200732698947","Type":"ContainerDied","Data":"837efdfdc8dc7f1b38914a9f78f524e3b43d9b14fcb059c384f701b1842cc98f"} Oct 30 17:03:56.595099 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.595057 4861 scope.go:122] "RemoveContainer" containerID="c8ab7e8f793235e335a8e420a1511e1969c8ff5518bebb748cf58b52e2559a47" Oct 30 17:03:56.596706 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.596680 4861 generic.go:356] "Generic (PLEG): container finished" podID="0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" containerID="781ea3197b9ec394596c212d5cf9dbfa31f7b04a98cfd2aadac157658ec0c650" exitCode=0 Oct 30 17:03:56.596857 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.596757 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" event={"ID":"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7","Type":"ContainerDied","Data":"781ea3197b9ec394596c212d5cf9dbfa31f7b04a98cfd2aadac157658ec0c650"} Oct 30 17:03:56.596857 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.596810 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" event={"ID":"0185f6c0-b86f-43ee-aab2-1c1d3728c7e7","Type":"ContainerDied","Data":"f13019b114755d22d944a7f7ecbd919c54d25cc11945fcfbb0d575096e38ecd7"} Oct 30 17:03:56.596857 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.596821 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn" Oct 30 17:03:56.608604 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.608585 4861 scope.go:122] "RemoveContainer" containerID="c8ab7e8f793235e335a8e420a1511e1969c8ff5518bebb748cf58b52e2559a47" Oct 30 17:03:56.608908 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:03:56.608891 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ab7e8f793235e335a8e420a1511e1969c8ff5518bebb748cf58b52e2559a47\": container with ID starting with c8ab7e8f793235e335a8e420a1511e1969c8ff5518bebb748cf58b52e2559a47 not found: ID does not exist" containerID="c8ab7e8f793235e335a8e420a1511e1969c8ff5518bebb748cf58b52e2559a47" Oct 30 17:03:56.608974 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.608916 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ab7e8f793235e335a8e420a1511e1969c8ff5518bebb748cf58b52e2559a47"} err="failed to get container status \"c8ab7e8f793235e335a8e420a1511e1969c8ff5518bebb748cf58b52e2559a47\": rpc error: code = NotFound desc = could not find container \"c8ab7e8f793235e335a8e420a1511e1969c8ff5518bebb748cf58b52e2559a47\": container with ID starting with c8ab7e8f793235e335a8e420a1511e1969c8ff5518bebb748cf58b52e2559a47 not found: ID does not exist" Oct 30 17:03:56.608974 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.608933 4861 scope.go:122] "RemoveContainer" containerID="781ea3197b9ec394596c212d5cf9dbfa31f7b04a98cfd2aadac157658ec0c650" Oct 30 17:03:56.620270 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.620252 4861 scope.go:122] "RemoveContainer" containerID="781ea3197b9ec394596c212d5cf9dbfa31f7b04a98cfd2aadac157658ec0c650" Oct 30 17:03:56.620609 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:03:56.620585 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"781ea3197b9ec394596c212d5cf9dbfa31f7b04a98cfd2aadac157658ec0c650\": container with ID starting with 781ea3197b9ec394596c212d5cf9dbfa31f7b04a98cfd2aadac157658ec0c650 not found: ID does not exist" containerID="781ea3197b9ec394596c212d5cf9dbfa31f7b04a98cfd2aadac157658ec0c650" Oct 30 17:03:56.620701 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.620614 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"781ea3197b9ec394596c212d5cf9dbfa31f7b04a98cfd2aadac157658ec0c650"} err="failed to get container status \"781ea3197b9ec394596c212d5cf9dbfa31f7b04a98cfd2aadac157658ec0c650\": rpc error: code = NotFound desc = could not find container \"781ea3197b9ec394596c212d5cf9dbfa31f7b04a98cfd2aadac157658ec0c650\": container with ID starting with 781ea3197b9ec394596c212d5cf9dbfa31f7b04a98cfd2aadac157658ec0c650 not found: ID does not exist" Oct 30 17:03:56.649828 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.649776 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq"] Oct 30 17:03:56.662374 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.662349 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-ckwrq"] Oct 30 17:03:56.691026 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.690996 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn"] Oct 30 17:03:56.697531 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:56.697509 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-81733d-openshift-default-688ffd6987-rn7sn"] Oct 30 17:03:57.808955 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:57.808922 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" path="/var/lib/kubelet/pods/0185f6c0-b86f-43ee-aab2-1c1d3728c7e7/volumes" Oct 30 17:03:57.809422 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:57.809408 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8682959-6953-4597-9c34-200732698947" path="/var/lib/kubelet/pods/c8682959-6953-4597-9c34-200732698947/volumes" Oct 30 17:03:58.735426 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.735392 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc"] Oct 30 17:03:58.736165 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.736150 4861 cpu_manager.go:414] "containerMap: removing container" podUID="0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" containerName="istio-proxy" Oct 30 17:03:58.736217 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.736167 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" containerName="istio-proxy" Oct 30 17:03:58.736217 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.736194 4861 cpu_manager.go:414] "containerMap: removing container" podUID="c8682959-6953-4597-9c34-200732698947" containerName="istio-proxy" Oct 30 17:03:58.736217 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.736199 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8682959-6953-4597-9c34-200732698947" containerName="istio-proxy" Oct 30 17:03:58.736313 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.736304 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="0185f6c0-b86f-43ee-aab2-1c1d3728c7e7" containerName="istio-proxy" Oct 30 17:03:58.736352 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.736316 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8682959-6953-4597-9c34-200732698947" containerName="istio-proxy" Oct 30 17:03:58.741930 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.741909 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" Oct 30 17:03:58.760539 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.760511 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc"] Oct 30 17:03:58.767821 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.767759 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-9bb5b1\"" Oct 30 17:03:58.768078 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.767850 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-9bb5b1-dockercfg-68grk\"" Oct 30 17:03:58.868998 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.868963 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cjv5\" (UniqueName: \"kubernetes.io/projected/4ddd17b2-6142-4f07-97a6-8ac5db77cad3-kube-api-access-4cjv5\") pod \"payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc\" (UID: \"4ddd17b2-6142-4f07-97a6-8ac5db77cad3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" Oct 30 17:03:58.882158 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.879883 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ddd17b2-6142-4f07-97a6-8ac5db77cad3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc\" (UID: \"4ddd17b2-6142-4f07-97a6-8ac5db77cad3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" Oct 30 17:03:58.882158 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.872136 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk"] Oct 30 17:03:58.895924 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.895539 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" Oct 30 17:03:58.910094 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.910049 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk"] Oct 30 17:03:58.982183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.982151 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ddd17b2-6142-4f07-97a6-8ac5db77cad3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc\" (UID: \"4ddd17b2-6142-4f07-97a6-8ac5db77cad3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" Oct 30 17:03:58.982386 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.982247 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17e7f96-ac06-40a0-99dd-a486a538970b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk\" (UID: \"d17e7f96-ac06-40a0-99dd-a486a538970b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" Oct 30 17:03:58.982471 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.982411 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhtkh\" (UniqueName: \"kubernetes.io/projected/d17e7f96-ac06-40a0-99dd-a486a538970b-kube-api-access-hhtkh\") pod \"payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk\" (UID: \"d17e7f96-ac06-40a0-99dd-a486a538970b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" Oct 30 17:03:58.982540 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.982522 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cjv5\" (UniqueName: \"kubernetes.io/projected/4ddd17b2-6142-4f07-97a6-8ac5db77cad3-kube-api-access-4cjv5\") pod \"payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc\" (UID: \"4ddd17b2-6142-4f07-97a6-8ac5db77cad3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" Oct 30 17:03:58.983013 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:58.982980 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ddd17b2-6142-4f07-97a6-8ac5db77cad3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc\" (UID: \"4ddd17b2-6142-4f07-97a6-8ac5db77cad3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" Oct 30 17:03:59.006633 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.006570 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cjv5\" (UniqueName: \"kubernetes.io/projected/4ddd17b2-6142-4f07-97a6-8ac5db77cad3-kube-api-access-4cjv5\") pod \"payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc\" (UID: \"4ddd17b2-6142-4f07-97a6-8ac5db77cad3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" Oct 30 17:03:59.056665 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.056641 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" Oct 30 17:03:59.083590 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.083562 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17e7f96-ac06-40a0-99dd-a486a538970b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk\" (UID: \"d17e7f96-ac06-40a0-99dd-a486a538970b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" Oct 30 17:03:59.083753 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.083706 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhtkh\" (UniqueName: \"kubernetes.io/projected/d17e7f96-ac06-40a0-99dd-a486a538970b-kube-api-access-hhtkh\") pod \"payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk\" (UID: \"d17e7f96-ac06-40a0-99dd-a486a538970b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" Oct 30 17:03:59.084312 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.084293 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17e7f96-ac06-40a0-99dd-a486a538970b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk\" (UID: \"d17e7f96-ac06-40a0-99dd-a486a538970b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" Oct 30 17:03:59.108063 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.108010 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhtkh\" (UniqueName: \"kubernetes.io/projected/d17e7f96-ac06-40a0-99dd-a486a538970b-kube-api-access-hhtkh\") pod \"payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk\" (UID: \"d17e7f96-ac06-40a0-99dd-a486a538970b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" Oct 30 17:03:59.227526 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.227493 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" Oct 30 17:03:59.373051 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:03:59.373012 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ddd17b2_6142_4f07_97a6_8ac5db77cad3.slice/crio-2b274eaae2530178acc70ed8f22e0e687705ddb84dc02a62cc99a7f388a02e46 WatchSource:0}: Error finding container 2b274eaae2530178acc70ed8f22e0e687705ddb84dc02a62cc99a7f388a02e46: Status 404 returned error can't find the container with id 2b274eaae2530178acc70ed8f22e0e687705ddb84dc02a62cc99a7f388a02e46 Oct 30 17:03:59.374322 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.374291 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc"] Oct 30 17:03:59.539019 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.538984 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk"] Oct 30 17:03:59.539552 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:03:59.539529 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17e7f96_ac06_40a0_99dd_a486a538970b.slice/crio-52a806c4cf3bb94c233733e4adfbd657978124f9ab8bf287d6a9d4f5ed93ddde WatchSource:0}: Error finding container 52a806c4cf3bb94c233733e4adfbd657978124f9ab8bf287d6a9d4f5ed93ddde: Status 404 returned error can't find the container with id 52a806c4cf3bb94c233733e4adfbd657978124f9ab8bf287d6a9d4f5ed93ddde Oct 30 17:03:59.625186 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.625155 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" event={"ID":"d17e7f96-ac06-40a0-99dd-a486a538970b","Type":"ContainerStarted","Data":"4f7afc4d10012b35f8d00c8dc8dbcb6cb6cc2e75e9bea1e32584bd07d302e9ff"} Oct 30 17:03:59.625333 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.625216 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" event={"ID":"d17e7f96-ac06-40a0-99dd-a486a538970b","Type":"ContainerStarted","Data":"52a806c4cf3bb94c233733e4adfbd657978124f9ab8bf287d6a9d4f5ed93ddde"} Oct 30 17:03:59.625333 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.625273 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" Oct 30 17:03:59.627051 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.627020 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" event={"ID":"4ddd17b2-6142-4f07-97a6-8ac5db77cad3","Type":"ContainerStarted","Data":"8a4dd259e23c4398fd01999fb71efabae3886b6ab3a9d8e1ceab749092c690b9"} Oct 30 17:03:59.627173 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.627071 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" event={"ID":"4ddd17b2-6142-4f07-97a6-8ac5db77cad3","Type":"ContainerStarted","Data":"2b274eaae2530178acc70ed8f22e0e687705ddb84dc02a62cc99a7f388a02e46"} Oct 30 17:03:59.627173 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.627092 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" Oct 30 17:03:59.668068 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.668022 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" podStartSLOduration=1.668010837 podStartE2EDuration="1.668010837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:59.666637792 +0000 UTC m=+2470.009311830" watchObservedRunningTime="2026-10-30 17:03:59.668010837 +0000 UTC m=+2470.010684874" Oct 30 17:03:59.715372 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:03:59.715326 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" podStartSLOduration=1.715313863 podStartE2EDuration="1.715313863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:59.710596054 +0000 UTC m=+2470.053270093" watchObservedRunningTime="2026-10-30 17:03:59.715313863 +0000 UTC m=+2470.057987901" Oct 30 17:04:10.636980 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:10.636901 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" Oct 30 17:04:10.636980 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:10.636980 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" Oct 30 17:04:20.290094 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:04:20.290055 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:04:25.910828 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:25.910790 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85"] Oct 30 17:04:26.574707 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:26.574669 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw"] Oct 30 17:04:30.911651 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:30.911612 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" podUID="fbb605e3-c317-4b71-9262-9a27d0868d78" containerName="payload-processing" containerID="cri-o://6c16d5dea18fcb29bb071d8dbe395df0aa272580e6a013b4dbcce2d9e5e5c993" gracePeriod=25 Oct 30 17:04:31.192392 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.192362 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:04:31.338691 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.338660 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb605e3-c317-4b71-9262-9a27d0868d78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb605e3-c317-4b71-9262-9a27d0868d78-plugins-config-volume\") pod \"fbb605e3-c317-4b71-9262-9a27d0868d78\" (UID: \"fbb605e3-c317-4b71-9262-9a27d0868d78\") " Oct 30 17:04:31.338869 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.338698 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb605e3-c317-4b71-9262-9a27d0868d78-kube-api-access-m6lnz\" (UniqueName: \"kubernetes.io/projected/fbb605e3-c317-4b71-9262-9a27d0868d78-kube-api-access-m6lnz\") pod \"fbb605e3-c317-4b71-9262-9a27d0868d78\" (UID: \"fbb605e3-c317-4b71-9262-9a27d0868d78\") " Oct 30 17:04:31.338869 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.338782 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb605e3-c317-4b71-9262-9a27d0868d78-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbb605e3-c317-4b71-9262-9a27d0868d78-provider-credentials\") pod \"fbb605e3-c317-4b71-9262-9a27d0868d78\" (UID: \"fbb605e3-c317-4b71-9262-9a27d0868d78\") " Oct 30 17:04:31.338977 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.338913 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb605e3-c317-4b71-9262-9a27d0868d78-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbb605e3-c317-4b71-9262-9a27d0868d78-routing-overlay-volume\") pod \"fbb605e3-c317-4b71-9262-9a27d0868d78\" (UID: \"fbb605e3-c317-4b71-9262-9a27d0868d78\") " Oct 30 17:04:31.339148 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.339112 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb605e3-c317-4b71-9262-9a27d0868d78-plugins-config-volume" pod "fbb605e3-c317-4b71-9262-9a27d0868d78" (UID: "fbb605e3-c317-4b71-9262-9a27d0868d78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:31.339425 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.339396 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb605e3-c317-4b71-9262-9a27d0868d78-routing-overlay-volume" pod "fbb605e3-c317-4b71-9262-9a27d0868d78" (UID: "fbb605e3-c317-4b71-9262-9a27d0868d78"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:31.339759 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.339714 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbb605e3-c317-4b71-9262-9a27d0868d78-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:04:31.339902 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.339763 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbb605e3-c317-4b71-9262-9a27d0868d78-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:04:31.342092 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.341932 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb605e3-c317-4b71-9262-9a27d0868d78-kube-api-access-m6lnz" pod "fbb605e3-c317-4b71-9262-9a27d0868d78" (UID: "fbb605e3-c317-4b71-9262-9a27d0868d78"). InnerVolumeSpecName "kube-api-access-m6lnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:31.347022 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.346970 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb605e3-c317-4b71-9262-9a27d0868d78-provider-credentials" pod "fbb605e3-c317-4b71-9262-9a27d0868d78" (UID: "fbb605e3-c317-4b71-9262-9a27d0868d78"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:31.441003 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.440948 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6lnz\" (UniqueName: \"kubernetes.io/projected/fbb605e3-c317-4b71-9262-9a27d0868d78-kube-api-access-m6lnz\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:04:31.441003 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.440971 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbb605e3-c317-4b71-9262-9a27d0868d78-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:04:31.574983 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.574943 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" podUID="df60f910-7e4b-4d21-ad99-7631726cb9d9" containerName="payload-processing" containerID="cri-o://23d543bd1263f06a80c2aec436ec69cfb490a37d3e129636f4c13f5e9072a103" gracePeriod=25 Oct 30 17:04:31.839092 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.839063 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:04:31.945902 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.945811 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df60f910-7e4b-4d21-ad99-7631726cb9d9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df60f910-7e4b-4d21-ad99-7631726cb9d9-routing-overlay-volume\") pod \"df60f910-7e4b-4d21-ad99-7631726cb9d9\" (UID: \"df60f910-7e4b-4d21-ad99-7631726cb9d9\") " Oct 30 17:04:31.945902 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.945886 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df60f910-7e4b-4d21-ad99-7631726cb9d9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/df60f910-7e4b-4d21-ad99-7631726cb9d9-provider-credentials\") pod \"df60f910-7e4b-4d21-ad99-7631726cb9d9\" (UID: \"df60f910-7e4b-4d21-ad99-7631726cb9d9\") " Oct 30 17:04:31.946435 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.946064 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df60f910-7e4b-4d21-ad99-7631726cb9d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df60f910-7e4b-4d21-ad99-7631726cb9d9-plugins-config-volume\") pod \"df60f910-7e4b-4d21-ad99-7631726cb9d9\" (UID: \"df60f910-7e4b-4d21-ad99-7631726cb9d9\") " Oct 30 17:04:31.946435 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.946207 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df60f910-7e4b-4d21-ad99-7631726cb9d9-routing-overlay-volume" pod "df60f910-7e4b-4d21-ad99-7631726cb9d9" (UID: "df60f910-7e4b-4d21-ad99-7631726cb9d9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:31.946435 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.946214 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df60f910-7e4b-4d21-ad99-7631726cb9d9-kube-api-access-99lbc\" (UniqueName: \"kubernetes.io/projected/df60f910-7e4b-4d21-ad99-7631726cb9d9-kube-api-access-99lbc\") pod \"df60f910-7e4b-4d21-ad99-7631726cb9d9\" (UID: \"df60f910-7e4b-4d21-ad99-7631726cb9d9\") " Oct 30 17:04:31.947220 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.946973 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df60f910-7e4b-4d21-ad99-7631726cb9d9-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:04:31.949002 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.948973 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df60f910-7e4b-4d21-ad99-7631726cb9d9-plugins-config-volume" pod "df60f910-7e4b-4d21-ad99-7631726cb9d9" (UID: "df60f910-7e4b-4d21-ad99-7631726cb9d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:31.949660 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.949633 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df60f910-7e4b-4d21-ad99-7631726cb9d9-kube-api-access-99lbc" pod "df60f910-7e4b-4d21-ad99-7631726cb9d9" (UID: "df60f910-7e4b-4d21-ad99-7631726cb9d9"). InnerVolumeSpecName "kube-api-access-99lbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:31.950996 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.950973 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df60f910-7e4b-4d21-ad99-7631726cb9d9-provider-credentials" pod "df60f910-7e4b-4d21-ad99-7631726cb9d9" (UID: "df60f910-7e4b-4d21-ad99-7631726cb9d9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:31.959623 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.959596 4861 generic.go:356] "Generic (PLEG): container finished" podID="df60f910-7e4b-4d21-ad99-7631726cb9d9" containerID="23d543bd1263f06a80c2aec436ec69cfb490a37d3e129636f4c13f5e9072a103" exitCode=0 Oct 30 17:04:31.959733 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.959642 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" Oct 30 17:04:31.959733 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.959684 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" event={"ID":"df60f910-7e4b-4d21-ad99-7631726cb9d9","Type":"ContainerDied","Data":"23d543bd1263f06a80c2aec436ec69cfb490a37d3e129636f4c13f5e9072a103"} Oct 30 17:04:31.959856 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.959757 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw" event={"ID":"df60f910-7e4b-4d21-ad99-7631726cb9d9","Type":"ContainerDied","Data":"8472730cdbd77f15d99da15ff606d45465c09d917e40041f2905fb3be589b875"} Oct 30 17:04:31.959856 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.959789 4861 scope.go:122] "RemoveContainer" containerID="23d543bd1263f06a80c2aec436ec69cfb490a37d3e129636f4c13f5e9072a103" Oct 30 17:04:31.961451 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.961429 4861 generic.go:356] "Generic (PLEG): container finished" podID="fbb605e3-c317-4b71-9262-9a27d0868d78" containerID="6c16d5dea18fcb29bb071d8dbe395df0aa272580e6a013b4dbcce2d9e5e5c993" exitCode=0 Oct 30 17:04:31.961560 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.961543 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" event={"ID":"fbb605e3-c317-4b71-9262-9a27d0868d78","Type":"ContainerDied","Data":"6c16d5dea18fcb29bb071d8dbe395df0aa272580e6a013b4dbcce2d9e5e5c993"} Oct 30 17:04:31.961615 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.961584 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" event={"ID":"fbb605e3-c317-4b71-9262-9a27d0868d78","Type":"ContainerDied","Data":"2714646188f861f51478241b9e86bd98f83e5756dc6412b065869ded15ac0ef9"} Oct 30 17:04:31.961615 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.961551 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85" Oct 30 17:04:31.975154 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.975138 4861 scope.go:122] "RemoveContainer" containerID="23d543bd1263f06a80c2aec436ec69cfb490a37d3e129636f4c13f5e9072a103" Oct 30 17:04:31.975443 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:04:31.975424 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d543bd1263f06a80c2aec436ec69cfb490a37d3e129636f4c13f5e9072a103\": container with ID starting with 23d543bd1263f06a80c2aec436ec69cfb490a37d3e129636f4c13f5e9072a103 not found: ID does not exist" containerID="23d543bd1263f06a80c2aec436ec69cfb490a37d3e129636f4c13f5e9072a103" Oct 30 17:04:31.975523 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.975455 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d543bd1263f06a80c2aec436ec69cfb490a37d3e129636f4c13f5e9072a103"} err="failed to get container status \"23d543bd1263f06a80c2aec436ec69cfb490a37d3e129636f4c13f5e9072a103\": rpc error: code = NotFound desc = could not find container \"23d543bd1263f06a80c2aec436ec69cfb490a37d3e129636f4c13f5e9072a103\": container with ID starting with 23d543bd1263f06a80c2aec436ec69cfb490a37d3e129636f4c13f5e9072a103 not found: ID does not exist" Oct 30 17:04:31.975523 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.975479 4861 scope.go:122] "RemoveContainer" containerID="6c16d5dea18fcb29bb071d8dbe395df0aa272580e6a013b4dbcce2d9e5e5c993" Oct 30 17:04:31.984849 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.984833 4861 scope.go:122] "RemoveContainer" containerID="6c16d5dea18fcb29bb071d8dbe395df0aa272580e6a013b4dbcce2d9e5e5c993" Oct 30 17:04:31.985122 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:04:31.985105 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c16d5dea18fcb29bb071d8dbe395df0aa272580e6a013b4dbcce2d9e5e5c993\": container with ID starting with 6c16d5dea18fcb29bb071d8dbe395df0aa272580e6a013b4dbcce2d9e5e5c993 not found: ID does not exist" containerID="6c16d5dea18fcb29bb071d8dbe395df0aa272580e6a013b4dbcce2d9e5e5c993" Oct 30 17:04:31.985174 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:31.985133 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c16d5dea18fcb29bb071d8dbe395df0aa272580e6a013b4dbcce2d9e5e5c993"} err="failed to get container status \"6c16d5dea18fcb29bb071d8dbe395df0aa272580e6a013b4dbcce2d9e5e5c993\": rpc error: code = NotFound desc = could not find container \"6c16d5dea18fcb29bb071d8dbe395df0aa272580e6a013b4dbcce2d9e5e5c993\": container with ID starting with 6c16d5dea18fcb29bb071d8dbe395df0aa272580e6a013b4dbcce2d9e5e5c993 not found: ID does not exist" Oct 30 17:04:32.017212 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:32.017178 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw"] Oct 30 17:04:32.020786 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:32.020758 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6d44h2qpw"] Oct 30 17:04:32.048285 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:32.048251 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99lbc\" (UniqueName: \"kubernetes.io/projected/df60f910-7e4b-4d21-ad99-7631726cb9d9-kube-api-access-99lbc\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:04:32.048285 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:32.048282 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df60f910-7e4b-4d21-ad99-7631726cb9d9-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:04:32.048434 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:32.048305 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df60f910-7e4b-4d21-ad99-7631726cb9d9-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:04:32.052032 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:32.052008 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85"] Oct 30 17:04:32.057982 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:32.057962 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-abda73/payload-processing-external-model-e2e-extmod-b-abda73-6cbcscv85"] Oct 30 17:04:33.808544 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:33.808509 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df60f910-7e4b-4d21-ad99-7631726cb9d9" path="/var/lib/kubelet/pods/df60f910-7e4b-4d21-ad99-7631726cb9d9/volumes" Oct 30 17:04:33.808979 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:33.808963 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb605e3-c317-4b71-9262-9a27d0868d78" path="/var/lib/kubelet/pods/fbb605e3-c317-4b71-9262-9a27d0868d78/volumes" Oct 30 17:04:35.638312 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.638281 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf"] Oct 30 17:04:35.639073 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.639058 4861 cpu_manager.go:414] "containerMap: removing container" podUID="fbb605e3-c317-4b71-9262-9a27d0868d78" containerName="payload-processing" Oct 30 17:04:35.639125 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.639074 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb605e3-c317-4b71-9262-9a27d0868d78" containerName="payload-processing" Oct 30 17:04:35.639125 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.639087 4861 cpu_manager.go:414] "containerMap: removing container" podUID="df60f910-7e4b-4d21-ad99-7631726cb9d9" containerName="payload-processing" Oct 30 17:04:35.639125 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.639092 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df60f910-7e4b-4d21-ad99-7631726cb9d9" containerName="payload-processing" Oct 30 17:04:35.639231 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.639192 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb605e3-c317-4b71-9262-9a27d0868d78" containerName="payload-processing" Oct 30 17:04:35.639231 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.639205 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="df60f910-7e4b-4d21-ad99-7631726cb9d9" containerName="payload-processing" Oct 30 17:04:35.644500 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.644485 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:04:35.667438 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.667412 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf"] Oct 30 17:04:35.671459 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.671437 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-9bb5b1-serving-cert\"" Oct 30 17:04:35.792882 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.792852 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f181e5-e3d0-42e1-a509-b92d3658a9b5-metrics-certs\") pod \"maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf\" (UID: \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:04:35.793074 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.792967 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63f181e5-e3d0-42e1-a509-b92d3658a9b5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf\" (UID: \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:04:35.793152 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.793098 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxrds\" (UniqueName: \"kubernetes.io/projected/63f181e5-e3d0-42e1-a509-b92d3658a9b5-kube-api-access-rxrds\") pod \"maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf\" (UID: \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:04:35.793248 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.793230 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f181e5-e3d0-42e1-a509-b92d3658a9b5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf\" (UID: \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:04:35.894769 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.894662 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxrds\" (UniqueName: \"kubernetes.io/projected/63f181e5-e3d0-42e1-a509-b92d3658a9b5-kube-api-access-rxrds\") pod \"maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf\" (UID: \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:04:35.894932 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.894803 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f181e5-e3d0-42e1-a509-b92d3658a9b5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf\" (UID: \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:04:35.894991 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.894966 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f181e5-e3d0-42e1-a509-b92d3658a9b5-metrics-certs\") pod \"maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf\" (UID: \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:04:35.895144 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.895125 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63f181e5-e3d0-42e1-a509-b92d3658a9b5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf\" (UID: \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:04:35.895556 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.895531 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f181e5-e3d0-42e1-a509-b92d3658a9b5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf\" (UID: \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:04:35.897644 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.897620 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63f181e5-e3d0-42e1-a509-b92d3658a9b5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf\" (UID: \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:04:35.897781 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.897697 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f181e5-e3d0-42e1-a509-b92d3658a9b5-metrics-certs\") pod \"maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf\" (UID: \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:04:35.919716 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.919690 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxrds\" (UniqueName: \"kubernetes.io/projected/63f181e5-e3d0-42e1-a509-b92d3658a9b5-kube-api-access-rxrds\") pod \"maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf\" (UID: \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:04:35.961436 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:35.961411 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:04:36.273348 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:36.273318 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf"] Oct 30 17:04:36.274022 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:04:36.273995 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f181e5_e3d0_42e1_a509_b92d3658a9b5.slice/crio-7a3609e8dfc078d903c6be4d58ced0f1e0f2836da27da12884c1613f17a5c1ba WatchSource:0}: Error finding container 7a3609e8dfc078d903c6be4d58ced0f1e0f2836da27da12884c1613f17a5c1ba: Status 404 returned error can't find the container with id 7a3609e8dfc078d903c6be4d58ced0f1e0f2836da27da12884c1613f17a5c1ba Oct 30 17:04:37.001670 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:37.001631 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" event={"ID":"63f181e5-e3d0-42e1-a509-b92d3658a9b5","Type":"ContainerStarted","Data":"9d938e0f89e530d2421a639a6b61afd5f2dcf2450ecbcdf098d4aa1313262a34"} Oct 30 17:04:37.002202 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:37.001698 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" event={"ID":"63f181e5-e3d0-42e1-a509-b92d3658a9b5","Type":"ContainerStarted","Data":"7a3609e8dfc078d903c6be4d58ced0f1e0f2836da27da12884c1613f17a5c1ba"} Oct 30 17:04:37.002202 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:37.001734 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:04:37.048836 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:37.048747 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" podStartSLOduration=1.533353313 podStartE2EDuration="2.048712674s" totalImagesPullingTime="515.359361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:37.047210301 +0000 UTC m=+2507.389884339" watchObservedRunningTime="2026-10-30 17:04:37.048712674 +0000 UTC m=+2507.391386711" Oct 30 17:04:43.015572 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:43.015540 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:04:50.314061 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:04:50.314013 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:04:53.113231 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:53.113196 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq"] Oct 30 17:04:58.113858 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:58.113821 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" podUID="35fc2733-2e52-469c-a59f-c7eae395d4cd" containerName="payload-processing" containerID="cri-o://a64ef0fa892f9991062c3f0c7231a9b90bd9dfb3380dda3aff3c62f578dc085e" gracePeriod=25 Oct 30 17:04:58.404283 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:58.404255 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:04:58.511881 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:58.511848 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35fc2733-2e52-469c-a59f-c7eae395d4cd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/35fc2733-2e52-469c-a59f-c7eae395d4cd-provider-credentials\") pod \"35fc2733-2e52-469c-a59f-c7eae395d4cd\" (UID: \"35fc2733-2e52-469c-a59f-c7eae395d4cd\") " Oct 30 17:04:58.512097 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:58.511897 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35fc2733-2e52-469c-a59f-c7eae395d4cd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35fc2733-2e52-469c-a59f-c7eae395d4cd-routing-overlay-volume\") pod \"35fc2733-2e52-469c-a59f-c7eae395d4cd\" (UID: \"35fc2733-2e52-469c-a59f-c7eae395d4cd\") " Oct 30 17:04:58.512097 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:58.511947 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35fc2733-2e52-469c-a59f-c7eae395d4cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35fc2733-2e52-469c-a59f-c7eae395d4cd-plugins-config-volume\") pod \"35fc2733-2e52-469c-a59f-c7eae395d4cd\" (UID: \"35fc2733-2e52-469c-a59f-c7eae395d4cd\") " Oct 30 17:04:58.512097 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:58.511969 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35fc2733-2e52-469c-a59f-c7eae395d4cd-kube-api-access-2mc55\" (UniqueName: \"kubernetes.io/projected/35fc2733-2e52-469c-a59f-c7eae395d4cd-kube-api-access-2mc55\") pod \"35fc2733-2e52-469c-a59f-c7eae395d4cd\" (UID: \"35fc2733-2e52-469c-a59f-c7eae395d4cd\") " Oct 30 17:04:58.512391 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:58.512366 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fc2733-2e52-469c-a59f-c7eae395d4cd-routing-overlay-volume" pod "35fc2733-2e52-469c-a59f-c7eae395d4cd" (UID: "35fc2733-2e52-469c-a59f-c7eae395d4cd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:58.512456 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:58.512402 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35fc2733-2e52-469c-a59f-c7eae395d4cd-plugins-config-volume" pod "35fc2733-2e52-469c-a59f-c7eae395d4cd" (UID: "35fc2733-2e52-469c-a59f-c7eae395d4cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:04:58.514415 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:58.514389 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fc2733-2e52-469c-a59f-c7eae395d4cd-provider-credentials" pod "35fc2733-2e52-469c-a59f-c7eae395d4cd" (UID: "35fc2733-2e52-469c-a59f-c7eae395d4cd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:58.514523 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:58.514425 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35fc2733-2e52-469c-a59f-c7eae395d4cd-kube-api-access-2mc55" pod "35fc2733-2e52-469c-a59f-c7eae395d4cd" (UID: "35fc2733-2e52-469c-a59f-c7eae395d4cd"). InnerVolumeSpecName "kube-api-access-2mc55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:58.614766 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:58.613839 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/35fc2733-2e52-469c-a59f-c7eae395d4cd-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:04:58.614766 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:58.613879 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/35fc2733-2e52-469c-a59f-c7eae395d4cd-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:04:58.614766 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:58.613894 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35fc2733-2e52-469c-a59f-c7eae395d4cd-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:04:58.614766 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:58.613912 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mc55\" (UniqueName: \"kubernetes.io/projected/35fc2733-2e52-469c-a59f-c7eae395d4cd-kube-api-access-2mc55\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:04:59.182989 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:59.182956 4861 generic.go:356] "Generic (PLEG): container finished" podID="35fc2733-2e52-469c-a59f-c7eae395d4cd" containerID="a64ef0fa892f9991062c3f0c7231a9b90bd9dfb3380dda3aff3c62f578dc085e" exitCode=0 Oct 30 17:04:59.183536 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:59.183002 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" Oct 30 17:04:59.183536 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:59.183047 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" event={"ID":"35fc2733-2e52-469c-a59f-c7eae395d4cd","Type":"ContainerDied","Data":"a64ef0fa892f9991062c3f0c7231a9b90bd9dfb3380dda3aff3c62f578dc085e"} Oct 30 17:04:59.183536 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:59.183114 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq" event={"ID":"35fc2733-2e52-469c-a59f-c7eae395d4cd","Type":"ContainerDied","Data":"cee8957f629a07363760cb4fb968888b870df984ab69a6e2258ec17373420ffb"} Oct 30 17:04:59.183536 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:59.183129 4861 scope.go:122] "RemoveContainer" containerID="a64ef0fa892f9991062c3f0c7231a9b90bd9dfb3380dda3aff3c62f578dc085e" Oct 30 17:04:59.194908 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:59.194891 4861 scope.go:122] "RemoveContainer" containerID="a64ef0fa892f9991062c3f0c7231a9b90bd9dfb3380dda3aff3c62f578dc085e" Oct 30 17:04:59.195195 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:04:59.195175 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a64ef0fa892f9991062c3f0c7231a9b90bd9dfb3380dda3aff3c62f578dc085e\": container with ID starting with a64ef0fa892f9991062c3f0c7231a9b90bd9dfb3380dda3aff3c62f578dc085e not found: ID does not exist" containerID="a64ef0fa892f9991062c3f0c7231a9b90bd9dfb3380dda3aff3c62f578dc085e" Oct 30 17:04:59.195267 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:59.195206 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a64ef0fa892f9991062c3f0c7231a9b90bd9dfb3380dda3aff3c62f578dc085e"} err="failed to get container status \"a64ef0fa892f9991062c3f0c7231a9b90bd9dfb3380dda3aff3c62f578dc085e\": rpc error: code = NotFound desc = could not find container \"a64ef0fa892f9991062c3f0c7231a9b90bd9dfb3380dda3aff3c62f578dc085e\": container with ID starting with a64ef0fa892f9991062c3f0c7231a9b90bd9dfb3380dda3aff3c62f578dc085e not found: ID does not exist" Oct 30 17:04:59.242964 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:59.242939 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq"] Oct 30 17:04:59.247163 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:59.247145 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-59d62bwnq"] Oct 30 17:04:59.808657 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:04:59.808612 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35fc2733-2e52-469c-a59f-c7eae395d4cd" path="/var/lib/kubelet/pods/35fc2733-2e52-469c-a59f-c7eae395d4cd/volumes" Oct 30 17:05:03.075343 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.075309 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx"] Oct 30 17:05:03.076085 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.076069 4861 cpu_manager.go:414] "containerMap: removing container" podUID="35fc2733-2e52-469c-a59f-c7eae395d4cd" containerName="payload-processing" Oct 30 17:05:03.076085 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.076085 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35fc2733-2e52-469c-a59f-c7eae395d4cd" containerName="payload-processing" Oct 30 17:05:03.076238 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.076229 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="35fc2733-2e52-469c-a59f-c7eae395d4cd" containerName="payload-processing" Oct 30 17:05:03.081379 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.081353 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:05:03.119277 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.119249 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"openshift-service-ca.crt\"" Oct 30 17:05:03.119582 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.119566 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"agc-cred-7c68f4c17b3a\"" Oct 30 17:05:03.119795 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.119779 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5c6b3c\"" Oct 30 17:05:03.119999 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.119985 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"payload-processing-external-model-e2e--22b3ecb4-dockercfg-6tsff\"" Oct 30 17:05:03.120167 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.120152 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"agc-cred-832ade4be86f\"" Oct 30 17:05:03.120368 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.120354 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"routing-overlay\"" Oct 30 17:05:03.120579 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.120567 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"kube-root-ca.crt\"" Oct 30 17:05:03.121281 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.121262 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"agc-cred-3e1593902499\"" Oct 30 17:05:03.126290 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.126248 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx\" (UID: \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:05:03.126407 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.126322 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx\" (UID: \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:05:03.126473 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.126422 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmstq\" (UniqueName: \"kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-kube-api-access-gmstq\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx\" (UID: \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:05:03.126514 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.126500 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx\" (UID: \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:05:03.139601 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.139577 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx"] Oct 30 17:05:03.227476 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.227439 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx\" (UID: \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:05:03.227676 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.227499 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx\" (UID: \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:05:03.227676 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.227561 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmstq\" (UniqueName: \"kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-kube-api-access-gmstq\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx\" (UID: \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:05:03.227676 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.227653 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx\" (UID: \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:05:03.228299 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.228269 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx\" (UID: \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:05:03.228415 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.228320 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx\" (UID: \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:05:03.231055 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.231032 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx\" (UID: \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:05:03.251279 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.251249 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmstq\" (UniqueName: \"kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-kube-api-access-gmstq\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx\" (UID: \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:05:03.399002 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.398924 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:05:03.713680 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:03.713645 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx"] Oct 30 17:05:03.715192 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:05:03.715162 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee4ae9f_daa3_4980_9fd7_57e4aa728157.slice/crio-22d7d93a0f76fb342f25bab703f276e403823e47527e12393403edd957816aaa WatchSource:0}: Error finding container 22d7d93a0f76fb342f25bab703f276e403823e47527e12393403edd957816aaa: Status 404 returned error can't find the container with id 22d7d93a0f76fb342f25bab703f276e403823e47527e12393403edd957816aaa Oct 30 17:05:04.232791 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:04.232749 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" event={"ID":"9ee4ae9f-daa3-4980-9fd7-57e4aa728157","Type":"ContainerStarted","Data":"885693b8a64c939dad2194c9185e0fe00eb561302c7313ed21f18b9ddadc1d60"} Oct 30 17:05:04.233226 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:04.232830 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:05:04.233226 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:04.232856 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" event={"ID":"9ee4ae9f-daa3-4980-9fd7-57e4aa728157","Type":"ContainerStarted","Data":"22d7d93a0f76fb342f25bab703f276e403823e47527e12393403edd957816aaa"} Oct 30 17:05:04.279034 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:04.278976 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" podStartSLOduration=1.278962162 podStartE2EDuration="1.278962162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:04.275162938 +0000 UTC m=+2534.617836976" watchObservedRunningTime="2026-10-30 17:05:04.278962162 +0000 UTC m=+2534.621636200" Oct 30 17:05:09.994073 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:09.994035 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z"] Oct 30 17:05:10.000350 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.000252 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:10.018709 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.018686 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z"] Oct 30 17:05:10.027329 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.027291 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"routing-overlay\"" Oct 30 17:05:10.027525 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.027327 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-3106d6\"" Oct 30 17:05:10.027673 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.027372 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"openshift-service-ca.crt\"" Oct 30 17:05:10.027777 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.027416 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"agc-cred-d8e0860828c0\"" Oct 30 17:05:10.027777 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.027468 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"payload-processing-external-model-e2e--ac3a616d-dockercfg-tvtqq\"" Oct 30 17:05:10.029032 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.028999 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"kube-root-ca.crt\"" Oct 30 17:05:10.057350 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.057319 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z\" (UID: \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:10.057482 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.057408 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z\" (UID: \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:10.057657 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.057629 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62dvm\" (UniqueName: \"kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-kube-api-access-62dvm\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z\" (UID: \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:10.057800 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.057781 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z\" (UID: \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:10.160756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.158857 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z\" (UID: \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:10.160756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.158930 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z\" (UID: \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:10.160756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.159015 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z\" (UID: \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:10.160756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.159206 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62dvm\" (UniqueName: \"kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-kube-api-access-62dvm\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z\" (UID: \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:10.166743 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.162372 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z\" (UID: \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:10.166743 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.163264 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z\" (UID: \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:10.181546 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.181516 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z\" (UID: \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:10.184524 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.184500 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62dvm\" (UniqueName: \"kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-kube-api-access-62dvm\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z\" (UID: \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:10.315765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.315661 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:10.591687 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.591610 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k"] Oct 30 17:05:10.597977 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.597949 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:10.614057 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.614028 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"agc-cred-167c20cb1fdd\"" Oct 30 17:05:10.618292 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.618265 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k"] Oct 30 17:05:10.626495 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.626468 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"payload-processing-external-model-e2e--5c3004c8-dockercfg-bwzsm\"" Oct 30 17:05:10.645871 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.645589 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z"] Oct 30 17:05:10.666886 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.666663 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zvx4\" (UniqueName: \"kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-kube-api-access-7zvx4\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k\" (UID: \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:10.667010 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.666895 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3585c31-63f9-4ffe-a69f-1c603e79baf6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k\" (UID: \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:10.667010 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.666927 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3585c31-63f9-4ffe-a69f-1c603e79baf6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k\" (UID: \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:10.667010 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.666943 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k\" (UID: \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:10.768063 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.768023 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3585c31-63f9-4ffe-a69f-1c603e79baf6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k\" (UID: \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:10.768235 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.768077 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3585c31-63f9-4ffe-a69f-1c603e79baf6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k\" (UID: \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:10.768235 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.768100 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k\" (UID: \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:10.768235 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.768216 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zvx4\" (UniqueName: \"kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-kube-api-access-7zvx4\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k\" (UID: \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:10.768782 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.768741 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3585c31-63f9-4ffe-a69f-1c603e79baf6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k\" (UID: \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:10.769064 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.768809 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3585c31-63f9-4ffe-a69f-1c603e79baf6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k\" (UID: \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:10.770792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.770769 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k\" (UID: \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:10.791905 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.791883 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zvx4\" (UniqueName: \"kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-kube-api-access-7zvx4\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k\" (UID: \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:10.914229 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:10.914146 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:11.230439 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:11.230410 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k"] Oct 30 17:05:11.230987 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:05:11.230963 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3585c31_63f9_4ffe_a69f_1c603e79baf6.slice/crio-f63c407e4a27752870262a3fe3fa9b70d5c0c45a51b66f6ff98cbb42c9c4a900 WatchSource:0}: Error finding container f63c407e4a27752870262a3fe3fa9b70d5c0c45a51b66f6ff98cbb42c9c4a900: Status 404 returned error can't find the container with id f63c407e4a27752870262a3fe3fa9b70d5c0c45a51b66f6ff98cbb42c9c4a900 Oct 30 17:05:11.291305 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:11.291245 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" event={"ID":"60c6da13-b7a1-4db3-a854-b93e83b9e5cc","Type":"ContainerStarted","Data":"d540604a3011c4312cf2012c6f5ca730f9e934e4062ea2be0e36771d2e185c25"} Oct 30 17:05:11.291494 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:11.291338 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" event={"ID":"60c6da13-b7a1-4db3-a854-b93e83b9e5cc","Type":"ContainerStarted","Data":"1ce6de2dbe76833e6dee077b0072d13be0b486f0138ffc065fecc32448027c12"} Oct 30 17:05:11.291494 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:11.291476 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:11.293048 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:11.293024 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" event={"ID":"e3585c31-63f9-4ffe-a69f-1c603e79baf6","Type":"ContainerStarted","Data":"f63c407e4a27752870262a3fe3fa9b70d5c0c45a51b66f6ff98cbb42c9c4a900"} Oct 30 17:05:11.338760 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:11.338659 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" podStartSLOduration=2.33864507 podStartE2EDuration="2.33864507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:11.334344315 +0000 UTC m=+2541.677018354" watchObservedRunningTime="2026-10-30 17:05:11.33864507 +0000 UTC m=+2541.681319144" Oct 30 17:05:11.376798 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:11.376768 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3106d6/agc-cred-d8e0860828c0: secret "agc-cred-d8e0860828c0" not found Oct 30 17:05:11.376798 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:11.376797 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z: secret "agc-cred-d8e0860828c0" not found Oct 30 17:05:11.377039 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:11.376863 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-provider-credentials podName:60c6da13-b7a1-4db3-a854-b93e83b9e5cc nodeName:}" failed. No retries permitted until 2026-10-30 17:05:11.876841053 +0000 UTC m=+2542.219515092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" (UID: "60c6da13-b7a1-4db3-a854-b93e83b9e5cc") : secret "agc-cred-d8e0860828c0" not found Oct 30 17:05:11.898217 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:11.898181 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3106d6/agc-cred-d8e0860828c0: secret "agc-cred-d8e0860828c0" not found Oct 30 17:05:11.898217 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:11.898215 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z: secret "agc-cred-d8e0860828c0" not found Oct 30 17:05:11.898457 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:11.898276 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-provider-credentials podName:60c6da13-b7a1-4db3-a854-b93e83b9e5cc nodeName:}" failed. No retries permitted until 2026-10-30 17:05:12.898255705 +0000 UTC m=+2543.240929744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" (UID: "60c6da13-b7a1-4db3-a854-b93e83b9e5cc") : secret "agc-cred-d8e0860828c0" not found Oct 30 17:05:11.975132 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:11.975099 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z"] Oct 30 17:05:12.329591 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:12.329546 4861 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" secret="" err="secret \"payload-processing-external-model-e2e--ac3a616d-dockercfg-tvtqq\" not found" Oct 30 17:05:12.333761 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:12.330904 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" event={"ID":"e3585c31-63f9-4ffe-a69f-1c603e79baf6","Type":"ContainerStarted","Data":"9d5a8808784be7e28a7fcd75d47f42bbd01a5cfdb316cbd689c4ae67f2cf8d60"} Oct 30 17:05:12.333761 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:12.331187 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:12.375911 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:12.375856 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" podStartSLOduration=2.375840504 podStartE2EDuration="2.375840504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:12.37454963 +0000 UTC m=+2542.717223673" watchObservedRunningTime="2026-10-30 17:05:12.375840504 +0000 UTC m=+2542.718514543" Oct 30 17:05:12.506518 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:12.506483 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5758b7/agc-cred-167c20cb1fdd: secret "agc-cred-167c20cb1fdd" not found Oct 30 17:05:12.506518 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:12.506517 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k: secret "agc-cred-167c20cb1fdd" not found Oct 30 17:05:12.506771 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:12.506574 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-provider-credentials podName:e3585c31-63f9-4ffe-a69f-1c603e79baf6 nodeName:}" failed. No retries permitted until 2026-10-30 17:05:13.006555021 +0000 UTC m=+2543.349229054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" (UID: "e3585c31-63f9-4ffe-a69f-1c603e79baf6") : secret "agc-cred-167c20cb1fdd" not found Oct 30 17:05:12.506771 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:12.506596 4861 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-plugins-e2e-extmod-a-3106d6: configmap "payload-processing-external-model-plugins-e2e-extmod-a-3106d6" not found Oct 30 17:05:12.506771 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:12.506651 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-plugins-config-volume podName:60c6da13-b7a1-4db3-a854-b93e83b9e5cc nodeName:}" failed. No retries permitted until 2026-10-30 17:05:13.006635267 +0000 UTC m=+2543.349309288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" (UID: "60c6da13-b7a1-4db3-a854-b93e83b9e5cc") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-3106d6" not found Oct 30 17:05:12.910262 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:12.910233 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3106d6/agc-cred-d8e0860828c0: secret "agc-cred-d8e0860828c0" not found Oct 30 17:05:12.910262 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:12.910258 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z: secret "agc-cred-d8e0860828c0" not found Oct 30 17:05:12.910474 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:12.910311 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-provider-credentials podName:60c6da13-b7a1-4db3-a854-b93e83b9e5cc nodeName:}" failed. No retries permitted until 2026-10-30 17:05:14.910294642 +0000 UTC m=+2545.252968660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" (UID: "60c6da13-b7a1-4db3-a854-b93e83b9e5cc") : secret "agc-cred-d8e0860828c0" not found Oct 30 17:05:13.011566 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:13.011530 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5758b7/agc-cred-167c20cb1fdd: secret "agc-cred-167c20cb1fdd" not found Oct 30 17:05:13.011566 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:13.011566 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k: secret "agc-cred-167c20cb1fdd" not found Oct 30 17:05:13.011846 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:13.011612 4861 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-plugins-e2e-extmod-a-3106d6: configmap "payload-processing-external-model-plugins-e2e-extmod-a-3106d6" not found Oct 30 17:05:13.011846 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:13.011624 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-provider-credentials podName:e3585c31-63f9-4ffe-a69f-1c603e79baf6 nodeName:}" failed. No retries permitted until 2026-10-30 17:05:14.011606721 +0000 UTC m=+2544.354280741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" (UID: "e3585c31-63f9-4ffe-a69f-1c603e79baf6") : secret "agc-cred-167c20cb1fdd" not found Oct 30 17:05:13.011846 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:13.011688 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-plugins-config-volume podName:60c6da13-b7a1-4db3-a854-b93e83b9e5cc nodeName:}" failed. No retries permitted until 2026-10-30 17:05:14.011672499 +0000 UTC m=+2544.354346521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" (UID: "60c6da13-b7a1-4db3-a854-b93e83b9e5cc") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-3106d6" not found Oct 30 17:05:13.208066 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:13.207988 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k"] Oct 30 17:05:14.027895 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:14.027865 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5758b7/agc-cred-167c20cb1fdd: secret "agc-cred-167c20cb1fdd" not found Oct 30 17:05:14.027895 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:14.027895 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k: secret "agc-cred-167c20cb1fdd" not found Oct 30 17:05:14.028329 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:14.027952 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-provider-credentials podName:e3585c31-63f9-4ffe-a69f-1c603e79baf6 nodeName:}" failed. No retries permitted until 2026-10-30 17:05:16.027937056 +0000 UTC m=+2546.370611073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" (UID: "e3585c31-63f9-4ffe-a69f-1c603e79baf6") : secret "agc-cred-167c20cb1fdd" not found Oct 30 17:05:14.028329 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:14.027968 4861 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-plugins-e2e-extmod-a-3106d6: configmap "payload-processing-external-model-plugins-e2e-extmod-a-3106d6" not found Oct 30 17:05:14.028329 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:14.028025 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-plugins-config-volume podName:60c6da13-b7a1-4db3-a854-b93e83b9e5cc nodeName:}" failed. No retries permitted until 2026-10-30 17:05:16.028012339 +0000 UTC m=+2546.370686359 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" (UID: "60c6da13-b7a1-4db3-a854-b93e83b9e5cc") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-3106d6" not found Oct 30 17:05:14.940873 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:14.940841 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3106d6/agc-cred-d8e0860828c0: secret "agc-cred-d8e0860828c0" not found Oct 30 17:05:14.940873 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:14.940869 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z: secret "agc-cred-d8e0860828c0" not found Oct 30 17:05:14.941154 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:14.940936 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-provider-credentials podName:60c6da13-b7a1-4db3-a854-b93e83b9e5cc nodeName:}" failed. No retries permitted until 2026-10-30 17:05:18.940917226 +0000 UTC m=+2549.283591255 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" (UID: "60c6da13-b7a1-4db3-a854-b93e83b9e5cc") : secret "agc-cred-d8e0860828c0" not found Oct 30 17:05:15.244257 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:15.244230 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:05:16.056316 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:16.056283 4861 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-plugins-e2e-extmod-a-3106d6: configmap "payload-processing-external-model-plugins-e2e-extmod-a-3106d6" not found Oct 30 17:05:16.056516 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:16.056357 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-plugins-config-volume podName:60c6da13-b7a1-4db3-a854-b93e83b9e5cc nodeName:}" failed. No retries permitted until 2026-10-30 17:05:20.056339011 +0000 UTC m=+2550.399013050 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" (UID: "60c6da13-b7a1-4db3-a854-b93e83b9e5cc") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-3106d6" not found Oct 30 17:05:16.056516 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:16.056403 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5758b7/agc-cred-167c20cb1fdd: secret "agc-cred-167c20cb1fdd" not found Oct 30 17:05:16.056516 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:16.056426 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k: secret "agc-cred-167c20cb1fdd" not found Oct 30 17:05:16.056516 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:16.056472 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-provider-credentials podName:e3585c31-63f9-4ffe-a69f-1c603e79baf6 nodeName:}" failed. No retries permitted until 2026-10-30 17:05:20.05646039 +0000 UTC m=+2550.399134418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" (UID: "e3585c31-63f9-4ffe-a69f-1c603e79baf6") : secret "agc-cred-167c20cb1fdd" not found Oct 30 17:05:18.351508 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:18.351466 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" podUID="60c6da13-b7a1-4db3-a854-b93e83b9e5cc" containerName="payload-processing" containerID="cri-o://d540604a3011c4312cf2012c6f5ca730f9e934e4062ea2be0e36771d2e185c25" gracePeriod=25 Oct 30 17:05:18.663870 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:18.663838 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:18.795967 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:18.795936 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-provider-credentials\") pod \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\" (UID: \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\") " Oct 30 17:05:18.796160 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:18.795998 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-kube-api-access-62dvm\" (UniqueName: \"kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-kube-api-access-62dvm\") pod \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\" (UID: \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\") " Oct 30 17:05:18.796160 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:18.796104 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-plugins-config-volume\") pod \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\" (UID: \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\") " Oct 30 17:05:18.796281 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:18.796156 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-routing-overlay-volume\") pod \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\" (UID: \"60c6da13-b7a1-4db3-a854-b93e83b9e5cc\") " Oct 30 17:05:18.796545 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:18.796515 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-routing-overlay-volume" pod "60c6da13-b7a1-4db3-a854-b93e83b9e5cc" (UID: "60c6da13-b7a1-4db3-a854-b93e83b9e5cc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:18.796638 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:18.796535 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-plugins-config-volume" pod "60c6da13-b7a1-4db3-a854-b93e83b9e5cc" (UID: "60c6da13-b7a1-4db3-a854-b93e83b9e5cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:18.797233 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:18.797210 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:18.797273 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:18.797244 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:18.798267 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:18.798248 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-provider-credentials" pod "60c6da13-b7a1-4db3-a854-b93e83b9e5cc" (UID: "60c6da13-b7a1-4db3-a854-b93e83b9e5cc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:18.798473 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:18.798453 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-kube-api-access-62dvm" pod "60c6da13-b7a1-4db3-a854-b93e83b9e5cc" (UID: "60c6da13-b7a1-4db3-a854-b93e83b9e5cc"). InnerVolumeSpecName "kube-api-access-62dvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:18.898631 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:18.898559 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:18.898631 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:18.898588 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62dvm\" (UniqueName: \"kubernetes.io/projected/60c6da13-b7a1-4db3-a854-b93e83b9e5cc-kube-api-access-62dvm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:19.357475 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.357434 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" podUID="e3585c31-63f9-4ffe-a69f-1c603e79baf6" containerName="payload-processing" containerID="cri-o://9d5a8808784be7e28a7fcd75d47f42bbd01a5cfdb316cbd689c4ae67f2cf8d60" gracePeriod=25 Oct 30 17:05:19.396234 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.396201 4861 generic.go:356] "Generic (PLEG): container finished" podID="60c6da13-b7a1-4db3-a854-b93e83b9e5cc" containerID="d540604a3011c4312cf2012c6f5ca730f9e934e4062ea2be0e36771d2e185c25" exitCode=0 Oct 30 17:05:19.396334 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.396237 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" event={"ID":"60c6da13-b7a1-4db3-a854-b93e83b9e5cc","Type":"ContainerDied","Data":"d540604a3011c4312cf2012c6f5ca730f9e934e4062ea2be0e36771d2e185c25"} Oct 30 17:05:19.396334 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.396271 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" Oct 30 17:05:19.396334 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.396308 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z" event={"ID":"60c6da13-b7a1-4db3-a854-b93e83b9e5cc","Type":"ContainerDied","Data":"1ce6de2dbe76833e6dee077b0072d13be0b486f0138ffc065fecc32448027c12"} Oct 30 17:05:19.396334 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.396331 4861 scope.go:122] "RemoveContainer" containerID="d540604a3011c4312cf2012c6f5ca730f9e934e4062ea2be0e36771d2e185c25" Oct 30 17:05:19.474309 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.474282 4861 scope.go:122] "RemoveContainer" containerID="d540604a3011c4312cf2012c6f5ca730f9e934e4062ea2be0e36771d2e185c25" Oct 30 17:05:19.474650 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:19.474623 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d540604a3011c4312cf2012c6f5ca730f9e934e4062ea2be0e36771d2e185c25\": container with ID starting with d540604a3011c4312cf2012c6f5ca730f9e934e4062ea2be0e36771d2e185c25 not found: ID does not exist" containerID="d540604a3011c4312cf2012c6f5ca730f9e934e4062ea2be0e36771d2e185c25" Oct 30 17:05:19.474759 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.474662 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d540604a3011c4312cf2012c6f5ca730f9e934e4062ea2be0e36771d2e185c25"} err="failed to get container status \"d540604a3011c4312cf2012c6f5ca730f9e934e4062ea2be0e36771d2e185c25\": rpc error: code = NotFound desc = could not find container \"d540604a3011c4312cf2012c6f5ca730f9e934e4062ea2be0e36771d2e185c25\": container with ID starting with d540604a3011c4312cf2012c6f5ca730f9e934e4062ea2be0e36771d2e185c25 not found: ID does not exist" Oct 30 17:05:19.475837 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.475816 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z"] Oct 30 17:05:19.489635 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.489612 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-76cc9kv4z"] Oct 30 17:05:19.641786 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.641764 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:19.711226 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.711193 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-kube-api-access-7zvx4\" (UniqueName: \"kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-kube-api-access-7zvx4\") pod \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\" (UID: \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\") " Oct 30 17:05:19.711411 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.711298 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3585c31-63f9-4ffe-a69f-1c603e79baf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3585c31-63f9-4ffe-a69f-1c603e79baf6-plugins-config-volume\") pod \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\" (UID: \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\") " Oct 30 17:05:19.711485 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.711465 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3585c31-63f9-4ffe-a69f-1c603e79baf6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3585c31-63f9-4ffe-a69f-1c603e79baf6-routing-overlay-volume\") pod \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\" (UID: \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\") " Oct 30 17:05:19.711553 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.711505 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-provider-credentials\") pod \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\" (UID: \"e3585c31-63f9-4ffe-a69f-1c603e79baf6\") " Oct 30 17:05:19.712125 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.711845 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3585c31-63f9-4ffe-a69f-1c603e79baf6-routing-overlay-volume" pod "e3585c31-63f9-4ffe-a69f-1c603e79baf6" (UID: "e3585c31-63f9-4ffe-a69f-1c603e79baf6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:19.712125 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.712101 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e3585c31-63f9-4ffe-a69f-1c603e79baf6-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:19.712526 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.712504 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3585c31-63f9-4ffe-a69f-1c603e79baf6-plugins-config-volume" pod "e3585c31-63f9-4ffe-a69f-1c603e79baf6" (UID: "e3585c31-63f9-4ffe-a69f-1c603e79baf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:19.714571 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.714543 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-provider-credentials" pod "e3585c31-63f9-4ffe-a69f-1c603e79baf6" (UID: "e3585c31-63f9-4ffe-a69f-1c603e79baf6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:19.716873 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.716850 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-kube-api-access-7zvx4" pod "e3585c31-63f9-4ffe-a69f-1c603e79baf6" (UID: "e3585c31-63f9-4ffe-a69f-1c603e79baf6"). InnerVolumeSpecName "kube-api-access-7zvx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:19.810121 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.810079 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c6da13-b7a1-4db3-a854-b93e83b9e5cc" path="/var/lib/kubelet/pods/60c6da13-b7a1-4db3-a854-b93e83b9e5cc/volumes" Oct 30 17:05:19.813894 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.813863 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3585c31-63f9-4ffe-a69f-1c603e79baf6-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:19.813894 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.813895 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:19.814140 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:19.813909 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zvx4\" (UniqueName: \"kubernetes.io/projected/e3585c31-63f9-4ffe-a69f-1c603e79baf6-kube-api-access-7zvx4\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:20.298358 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:20.298319 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:05:20.405789 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:20.405757 4861 generic.go:356] "Generic (PLEG): container finished" podID="e3585c31-63f9-4ffe-a69f-1c603e79baf6" containerID="9d5a8808784be7e28a7fcd75d47f42bbd01a5cfdb316cbd689c4ae67f2cf8d60" exitCode=0 Oct 30 17:05:20.406224 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:20.405811 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" Oct 30 17:05:20.406224 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:20.405837 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" event={"ID":"e3585c31-63f9-4ffe-a69f-1c603e79baf6","Type":"ContainerDied","Data":"9d5a8808784be7e28a7fcd75d47f42bbd01a5cfdb316cbd689c4ae67f2cf8d60"} Oct 30 17:05:20.406224 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:20.405894 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k" event={"ID":"e3585c31-63f9-4ffe-a69f-1c603e79baf6","Type":"ContainerDied","Data":"f63c407e4a27752870262a3fe3fa9b70d5c0c45a51b66f6ff98cbb42c9c4a900"} Oct 30 17:05:20.406224 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:20.405920 4861 scope.go:122] "RemoveContainer" containerID="9d5a8808784be7e28a7fcd75d47f42bbd01a5cfdb316cbd689c4ae67f2cf8d60" Oct 30 17:05:20.421792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:20.421767 4861 scope.go:122] "RemoveContainer" containerID="9d5a8808784be7e28a7fcd75d47f42bbd01a5cfdb316cbd689c4ae67f2cf8d60" Oct 30 17:05:20.422070 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:20.422047 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5a8808784be7e28a7fcd75d47f42bbd01a5cfdb316cbd689c4ae67f2cf8d60\": container with ID starting with 9d5a8808784be7e28a7fcd75d47f42bbd01a5cfdb316cbd689c4ae67f2cf8d60 not found: ID does not exist" containerID="9d5a8808784be7e28a7fcd75d47f42bbd01a5cfdb316cbd689c4ae67f2cf8d60" Oct 30 17:05:20.422152 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:20.422081 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5a8808784be7e28a7fcd75d47f42bbd01a5cfdb316cbd689c4ae67f2cf8d60"} err="failed to get container status \"9d5a8808784be7e28a7fcd75d47f42bbd01a5cfdb316cbd689c4ae67f2cf8d60\": rpc error: code = NotFound desc = could not find container \"9d5a8808784be7e28a7fcd75d47f42bbd01a5cfdb316cbd689c4ae67f2cf8d60\": container with ID starting with 9d5a8808784be7e28a7fcd75d47f42bbd01a5cfdb316cbd689c4ae67f2cf8d60 not found: ID does not exist" Oct 30 17:05:20.475123 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:20.475095 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k"] Oct 30 17:05:20.488764 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:20.488711 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-9787lsz5k"] Oct 30 17:05:21.809207 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:21.809170 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3585c31-63f9-4ffe-a69f-1c603e79baf6" path="/var/lib/kubelet/pods/e3585c31-63f9-4ffe-a69f-1c603e79baf6/volumes" Oct 30 17:05:27.664504 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:27.664472 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6"] Oct 30 17:05:32.665141 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:32.665057 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" podUID="1536eff9-3744-4068-ae94-f2063461b26f" containerName="payload-processing" containerID="cri-o://d24954d430504ca1c05c8311ccfa363f4a2eaf67b9417b2db065f3ef812e2c0c" gracePeriod=25 Oct 30 17:05:32.950295 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:32.950267 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:05:33.097133 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.097103 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1536eff9-3744-4068-ae94-f2063461b26f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1536eff9-3744-4068-ae94-f2063461b26f-provider-credentials\") pod \"1536eff9-3744-4068-ae94-f2063461b26f\" (UID: \"1536eff9-3744-4068-ae94-f2063461b26f\") " Oct 30 17:05:33.097359 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.097141 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1536eff9-3744-4068-ae94-f2063461b26f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1536eff9-3744-4068-ae94-f2063461b26f-routing-overlay-volume\") pod \"1536eff9-3744-4068-ae94-f2063461b26f\" (UID: \"1536eff9-3744-4068-ae94-f2063461b26f\") " Oct 30 17:05:33.097359 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.097171 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1536eff9-3744-4068-ae94-f2063461b26f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1536eff9-3744-4068-ae94-f2063461b26f-plugins-config-volume\") pod \"1536eff9-3744-4068-ae94-f2063461b26f\" (UID: \"1536eff9-3744-4068-ae94-f2063461b26f\") " Oct 30 17:05:33.097359 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.097332 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1536eff9-3744-4068-ae94-f2063461b26f-kube-api-access-hl5pv\" (UniqueName: \"kubernetes.io/projected/1536eff9-3744-4068-ae94-f2063461b26f-kube-api-access-hl5pv\") pod \"1536eff9-3744-4068-ae94-f2063461b26f\" (UID: \"1536eff9-3744-4068-ae94-f2063461b26f\") " Oct 30 17:05:33.097552 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.097529 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1536eff9-3744-4068-ae94-f2063461b26f-routing-overlay-volume" pod "1536eff9-3744-4068-ae94-f2063461b26f" (UID: "1536eff9-3744-4068-ae94-f2063461b26f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:33.097660 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.097631 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1536eff9-3744-4068-ae94-f2063461b26f-plugins-config-volume" pod "1536eff9-3744-4068-ae94-f2063461b26f" (UID: "1536eff9-3744-4068-ae94-f2063461b26f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:33.097822 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.097803 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1536eff9-3744-4068-ae94-f2063461b26f-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:33.097877 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.097829 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1536eff9-3744-4068-ae94-f2063461b26f-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:33.099679 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.099651 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1536eff9-3744-4068-ae94-f2063461b26f-provider-credentials" pod "1536eff9-3744-4068-ae94-f2063461b26f" (UID: "1536eff9-3744-4068-ae94-f2063461b26f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:33.099823 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.099689 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1536eff9-3744-4068-ae94-f2063461b26f-kube-api-access-hl5pv" pod "1536eff9-3744-4068-ae94-f2063461b26f" (UID: "1536eff9-3744-4068-ae94-f2063461b26f"). InnerVolumeSpecName "kube-api-access-hl5pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:33.198883 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.198807 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1536eff9-3744-4068-ae94-f2063461b26f-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:33.198883 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.198834 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl5pv\" (UniqueName: \"kubernetes.io/projected/1536eff9-3744-4068-ae94-f2063461b26f-kube-api-access-hl5pv\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:33.539267 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.537245 4861 generic.go:356] "Generic (PLEG): container finished" podID="1536eff9-3744-4068-ae94-f2063461b26f" containerID="d24954d430504ca1c05c8311ccfa363f4a2eaf67b9417b2db065f3ef812e2c0c" exitCode=0 Oct 30 17:05:33.539267 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.537364 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" event={"ID":"1536eff9-3744-4068-ae94-f2063461b26f","Type":"ContainerDied","Data":"d24954d430504ca1c05c8311ccfa363f4a2eaf67b9417b2db065f3ef812e2c0c"} Oct 30 17:05:33.539267 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.537450 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" event={"ID":"1536eff9-3744-4068-ae94-f2063461b26f","Type":"ContainerDied","Data":"2fe79ac47113f7cc80cfbcd6a2e53f1684b57caf5aa8d787e7f0aa0ecece57a3"} Oct 30 17:05:33.539267 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.537481 4861 scope.go:122] "RemoveContainer" containerID="d24954d430504ca1c05c8311ccfa363f4a2eaf67b9417b2db065f3ef812e2c0c" Oct 30 17:05:33.539267 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.537650 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6" Oct 30 17:05:33.553175 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.553156 4861 scope.go:122] "RemoveContainer" containerID="d24954d430504ca1c05c8311ccfa363f4a2eaf67b9417b2db065f3ef812e2c0c" Oct 30 17:05:33.553442 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:33.553420 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d24954d430504ca1c05c8311ccfa363f4a2eaf67b9417b2db065f3ef812e2c0c\": container with ID starting with d24954d430504ca1c05c8311ccfa363f4a2eaf67b9417b2db065f3ef812e2c0c not found: ID does not exist" containerID="d24954d430504ca1c05c8311ccfa363f4a2eaf67b9417b2db065f3ef812e2c0c" Oct 30 17:05:33.553509 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.553455 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d24954d430504ca1c05c8311ccfa363f4a2eaf67b9417b2db065f3ef812e2c0c"} err="failed to get container status \"d24954d430504ca1c05c8311ccfa363f4a2eaf67b9417b2db065f3ef812e2c0c\": rpc error: code = NotFound desc = could not find container \"d24954d430504ca1c05c8311ccfa363f4a2eaf67b9417b2db065f3ef812e2c0c\": container with ID starting with d24954d430504ca1c05c8311ccfa363f4a2eaf67b9417b2db065f3ef812e2c0c not found: ID does not exist" Oct 30 17:05:33.625029 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.624988 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6"] Oct 30 17:05:33.636988 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.636965 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-85897x6q6"] Oct 30 17:05:33.809093 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:33.809015 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1536eff9-3744-4068-ae94-f2063461b26f" path="/var/lib/kubelet/pods/1536eff9-3744-4068-ae94-f2063461b26f/volumes" Oct 30 17:05:45.889771 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.889739 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m"] Oct 30 17:05:45.890538 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.890505 4861 cpu_manager.go:414] "containerMap: removing container" podUID="e3585c31-63f9-4ffe-a69f-1c603e79baf6" containerName="payload-processing" Oct 30 17:05:45.890538 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.890523 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3585c31-63f9-4ffe-a69f-1c603e79baf6" containerName="payload-processing" Oct 30 17:05:45.890702 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.890550 4861 cpu_manager.go:414] "containerMap: removing container" podUID="1536eff9-3744-4068-ae94-f2063461b26f" containerName="payload-processing" Oct 30 17:05:45.890702 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.890559 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1536eff9-3744-4068-ae94-f2063461b26f" containerName="payload-processing" Oct 30 17:05:45.890702 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.890639 4861 cpu_manager.go:414] "containerMap: removing container" podUID="60c6da13-b7a1-4db3-a854-b93e83b9e5cc" containerName="payload-processing" Oct 30 17:05:45.890702 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.890651 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c6da13-b7a1-4db3-a854-b93e83b9e5cc" containerName="payload-processing" Oct 30 17:05:45.890876 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.890822 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="1536eff9-3744-4068-ae94-f2063461b26f" containerName="payload-processing" Oct 30 17:05:45.890876 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.890851 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c6da13-b7a1-4db3-a854-b93e83b9e5cc" containerName="payload-processing" Oct 30 17:05:45.890876 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.890874 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3585c31-63f9-4ffe-a69f-1c603e79baf6" containerName="payload-processing" Oct 30 17:05:45.894818 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.894802 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:05:45.942814 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.937366 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"openshift-service-ca.crt\"" Oct 30 17:05:45.942814 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.937463 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"agc-cred-6be2aaced4b8\"" Oct 30 17:05:45.942814 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.937461 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"routing-overlay\"" Oct 30 17:05:45.942814 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.937536 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"kube-root-ca.crt\"" Oct 30 17:05:45.942814 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.937649 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-3106d6\"" Oct 30 17:05:45.942814 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.937653 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"payload-processing-external-model-e2e--ac3a616d-dockercfg-nw26z\"" Oct 30 17:05:45.942814 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.942093 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m"] Oct 30 17:05:45.976234 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.976188 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8w92\" (UniqueName: \"kubernetes.io/projected/aeb6353a-f1de-4669-afd7-a17f36d2a194-kube-api-access-m8w92\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m\" (UID: \"aeb6353a-f1de-4669-afd7-a17f36d2a194\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:05:45.976452 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.976338 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb6353a-f1de-4669-afd7-a17f36d2a194-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m\" (UID: \"aeb6353a-f1de-4669-afd7-a17f36d2a194\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:05:45.976452 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.976379 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aeb6353a-f1de-4669-afd7-a17f36d2a194-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m\" (UID: \"aeb6353a-f1de-4669-afd7-a17f36d2a194\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:05:45.976452 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:45.976404 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aeb6353a-f1de-4669-afd7-a17f36d2a194-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m\" (UID: \"aeb6353a-f1de-4669-afd7-a17f36d2a194\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:05:46.077820 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:46.077786 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8w92\" (UniqueName: \"kubernetes.io/projected/aeb6353a-f1de-4669-afd7-a17f36d2a194-kube-api-access-m8w92\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m\" (UID: \"aeb6353a-f1de-4669-afd7-a17f36d2a194\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:05:46.078000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:46.077924 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb6353a-f1de-4669-afd7-a17f36d2a194-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m\" (UID: \"aeb6353a-f1de-4669-afd7-a17f36d2a194\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:05:46.078000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:46.077957 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aeb6353a-f1de-4669-afd7-a17f36d2a194-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m\" (UID: \"aeb6353a-f1de-4669-afd7-a17f36d2a194\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:05:46.078000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:46.077980 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aeb6353a-f1de-4669-afd7-a17f36d2a194-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m\" (UID: \"aeb6353a-f1de-4669-afd7-a17f36d2a194\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:05:46.078762 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:46.078688 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aeb6353a-f1de-4669-afd7-a17f36d2a194-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m\" (UID: \"aeb6353a-f1de-4669-afd7-a17f36d2a194\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:05:46.078934 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:46.078885 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb6353a-f1de-4669-afd7-a17f36d2a194-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m\" (UID: \"aeb6353a-f1de-4669-afd7-a17f36d2a194\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:05:46.080851 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:46.080823 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aeb6353a-f1de-4669-afd7-a17f36d2a194-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m\" (UID: \"aeb6353a-f1de-4669-afd7-a17f36d2a194\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:05:46.106050 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:46.106021 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8w92\" (UniqueName: \"kubernetes.io/projected/aeb6353a-f1de-4669-afd7-a17f36d2a194-kube-api-access-m8w92\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m\" (UID: \"aeb6353a-f1de-4669-afd7-a17f36d2a194\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:05:46.213124 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:46.211629 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:05:46.536597 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:46.536567 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m"] Oct 30 17:05:46.538235 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:05:46.538205 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb6353a_f1de_4669_afd7_a17f36d2a194.slice/crio-2fcff230f78cf71e770fde49fd8d0c45bd16ab8306fac2cc83b2e5bce35ee6c1 WatchSource:0}: Error finding container 2fcff230f78cf71e770fde49fd8d0c45bd16ab8306fac2cc83b2e5bce35ee6c1: Status 404 returned error can't find the container with id 2fcff230f78cf71e770fde49fd8d0c45bd16ab8306fac2cc83b2e5bce35ee6c1 Oct 30 17:05:46.660783 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:46.660752 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" event={"ID":"aeb6353a-f1de-4669-afd7-a17f36d2a194","Type":"ContainerStarted","Data":"52cd4df6b164ec73cf9e89d3c32e2dfd96f90d4bc3feaa4cc544492617024f0e"} Oct 30 17:05:46.660887 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:46.660804 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" event={"ID":"aeb6353a-f1de-4669-afd7-a17f36d2a194","Type":"ContainerStarted","Data":"2fcff230f78cf71e770fde49fd8d0c45bd16ab8306fac2cc83b2e5bce35ee6c1"} Oct 30 17:05:46.660989 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:46.660970 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:05:46.703233 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:46.703189 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" podStartSLOduration=1.703176509 podStartE2EDuration="1.703176509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:46.703047789 +0000 UTC m=+2577.045721830" watchObservedRunningTime="2026-10-30 17:05:46.703176509 +0000 UTC m=+2577.045850548" Oct 30 17:05:48.166854 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:48.166819 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8"] Oct 30 17:05:48.167292 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:48.167058 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerName="istio-proxy" containerID="cri-o://0b4459e251555fed4b5e9f90878719cec951a5b8b5b641a7d5c8087eb027da97" gracePeriod=30 Oct 30 17:05:50.290665 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:50.290627 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:05:53.456459 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.456387 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 17:05:53.589324 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.589295 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-workload-certs\") pod \"db9b9641-1b2c-49b7-9169-59cf182f4900\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " Oct 30 17:05:53.589324 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.589329 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-envoy\") pod \"db9b9641-1b2c-49b7-9169-59cf182f4900\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " Oct 30 17:05:53.589549 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.589363 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db9b9641-1b2c-49b7-9169-59cf182f4900-kube-api-access-7w4tj\" (UniqueName: \"kubernetes.io/projected/db9b9641-1b2c-49b7-9169-59cf182f4900-kube-api-access-7w4tj\") pod \"db9b9641-1b2c-49b7-9169-59cf182f4900\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " Oct 30 17:05:53.589549 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.589410 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db9b9641-1b2c-49b7-9169-59cf182f4900-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9b9641-1b2c-49b7-9169-59cf182f4900-istiod-ca-cert\") pod \"db9b9641-1b2c-49b7-9169-59cf182f4900\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " Oct 30 17:05:53.589549 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.589534 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-data\") pod \"db9b9641-1b2c-49b7-9169-59cf182f4900\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " Oct 30 17:05:53.589779 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.589605 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-workload-certs" pod "db9b9641-1b2c-49b7-9169-59cf182f4900" (UID: "db9b9641-1b2c-49b7-9169-59cf182f4900"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:53.589779 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.589682 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-workload-socket\") pod \"db9b9641-1b2c-49b7-9169-59cf182f4900\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " Oct 30 17:05:53.589909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.589803 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-credential-socket\") pod \"db9b9641-1b2c-49b7-9169-59cf182f4900\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " Oct 30 17:05:53.589909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.589823 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-data" pod "db9b9641-1b2c-49b7-9169-59cf182f4900" (UID: "db9b9641-1b2c-49b7-9169-59cf182f4900"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:53.589909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.589838 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9b9641-1b2c-49b7-9169-59cf182f4900-istiod-ca-cert" pod "db9b9641-1b2c-49b7-9169-59cf182f4900" (UID: "db9b9641-1b2c-49b7-9169-59cf182f4900"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:05:53.589909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.589862 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-token\" (UniqueName: \"kubernetes.io/projected/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-token\") pod \"db9b9641-1b2c-49b7-9169-59cf182f4900\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " Oct 30 17:05:53.589909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.589906 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-podinfo\") pod \"db9b9641-1b2c-49b7-9169-59cf182f4900\" (UID: \"db9b9641-1b2c-49b7-9169-59cf182f4900\") " Oct 30 17:05:53.590176 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.589926 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-workload-socket" pod "db9b9641-1b2c-49b7-9169-59cf182f4900" (UID: "db9b9641-1b2c-49b7-9169-59cf182f4900"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:53.590176 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.590069 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-credential-socket" pod "db9b9641-1b2c-49b7-9169-59cf182f4900" (UID: "db9b9641-1b2c-49b7-9169-59cf182f4900"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:53.590508 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.590487 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:53.590595 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.590513 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db9b9641-1b2c-49b7-9169-59cf182f4900-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:53.590595 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.590526 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:53.590595 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.590539 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:53.590595 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.590554 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:53.591998 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.591969 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-envoy" pod "db9b9641-1b2c-49b7-9169-59cf182f4900" (UID: "db9b9641-1b2c-49b7-9169-59cf182f4900"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:53.592111 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.592013 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9b9641-1b2c-49b7-9169-59cf182f4900-kube-api-access-7w4tj" pod "db9b9641-1b2c-49b7-9169-59cf182f4900" (UID: "db9b9641-1b2c-49b7-9169-59cf182f4900"). InnerVolumeSpecName "kube-api-access-7w4tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:53.592756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.592707 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-podinfo" pod "db9b9641-1b2c-49b7-9169-59cf182f4900" (UID: "db9b9641-1b2c-49b7-9169-59cf182f4900"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:05:53.592855 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.592767 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-token" pod "db9b9641-1b2c-49b7-9169-59cf182f4900" (UID: "db9b9641-1b2c-49b7-9169-59cf182f4900"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:53.692565 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.692518 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:53.692565 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.692555 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w4tj\" (UniqueName: \"kubernetes.io/projected/db9b9641-1b2c-49b7-9169-59cf182f4900-kube-api-access-7w4tj\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:53.692565 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.692573 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:53.692915 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.692586 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db9b9641-1b2c-49b7-9169-59cf182f4900-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:05:53.716594 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.716521 4861 generic.go:356] "Generic (PLEG): container finished" podID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerID="0b4459e251555fed4b5e9f90878719cec951a5b8b5b641a7d5c8087eb027da97" exitCode=0 Oct 30 17:05:53.716594 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.716580 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" Oct 30 17:05:53.716788 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.716598 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" event={"ID":"db9b9641-1b2c-49b7-9169-59cf182f4900","Type":"ContainerDied","Data":"0b4459e251555fed4b5e9f90878719cec951a5b8b5b641a7d5c8087eb027da97"} Oct 30 17:05:53.716788 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.716658 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8" event={"ID":"db9b9641-1b2c-49b7-9169-59cf182f4900","Type":"ContainerDied","Data":"d8c8140130350aebb7735722d6a2d5c5550533ac387cc08cb6650feec5ad299c"} Oct 30 17:05:53.716788 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.716688 4861 scope.go:122] "RemoveContainer" containerID="0b4459e251555fed4b5e9f90878719cec951a5b8b5b641a7d5c8087eb027da97" Oct 30 17:05:53.730501 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.730478 4861 scope.go:122] "RemoveContainer" containerID="0b4459e251555fed4b5e9f90878719cec951a5b8b5b641a7d5c8087eb027da97" Oct 30 17:05:53.730829 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:05:53.730809 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4459e251555fed4b5e9f90878719cec951a5b8b5b641a7d5c8087eb027da97\": container with ID starting with 0b4459e251555fed4b5e9f90878719cec951a5b8b5b641a7d5c8087eb027da97 not found: ID does not exist" containerID="0b4459e251555fed4b5e9f90878719cec951a5b8b5b641a7d5c8087eb027da97" Oct 30 17:05:53.730885 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.730837 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4459e251555fed4b5e9f90878719cec951a5b8b5b641a7d5c8087eb027da97"} err="failed to get container status \"0b4459e251555fed4b5e9f90878719cec951a5b8b5b641a7d5c8087eb027da97\": rpc error: code = NotFound desc = could not find container \"0b4459e251555fed4b5e9f90878719cec951a5b8b5b641a7d5c8087eb027da97\": container with ID starting with 0b4459e251555fed4b5e9f90878719cec951a5b8b5b641a7d5c8087eb027da97 not found: ID does not exist" Oct 30 17:05:53.774988 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.774965 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8"] Oct 30 17:05:53.823000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:53.822921 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-6mvz8"] Oct 30 17:05:55.812545 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:55.812513 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" path="/var/lib/kubelet/pods/db9b9641-1b2c-49b7-9169-59cf182f4900/volumes" Oct 30 17:05:57.671044 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:05:57.671005 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:06:11.820164 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:11.820130 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj"] Oct 30 17:06:11.820908 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:11.820893 4861 cpu_manager.go:414] "containerMap: removing container" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerName="istio-proxy" Oct 30 17:06:11.820908 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:11.820909 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerName="istio-proxy" Oct 30 17:06:11.821008 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:11.821002 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="db9b9641-1b2c-49b7-9169-59cf182f4900" containerName="istio-proxy" Oct 30 17:06:11.826670 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:11.826650 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:06:11.847338 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:11.847296 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj"] Oct 30 17:06:11.856038 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:11.855990 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-43197b-serving-cert\"" Oct 30 17:06:11.959557 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:11.959530 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20eaed92-2c7a-4490-bc40-4efcb335b0bb-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj\" (UID: \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:06:11.959764 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:11.959690 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20eaed92-2c7a-4490-bc40-4efcb335b0bb-maas-api-tls\") pod \"maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj\" (UID: \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:06:11.959870 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:11.959768 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20eaed92-2c7a-4490-bc40-4efcb335b0bb-metrics-certs\") pod \"maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj\" (UID: \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:06:11.959870 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:11.959866 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcsrf\" (UniqueName: \"kubernetes.io/projected/20eaed92-2c7a-4490-bc40-4efcb335b0bb-kube-api-access-wcsrf\") pod \"maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj\" (UID: \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:06:12.061585 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:12.061544 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20eaed92-2c7a-4490-bc40-4efcb335b0bb-metrics-certs\") pod \"maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj\" (UID: \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:06:12.061815 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:12.061661 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcsrf\" (UniqueName: \"kubernetes.io/projected/20eaed92-2c7a-4490-bc40-4efcb335b0bb-kube-api-access-wcsrf\") pod \"maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj\" (UID: \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:06:12.061815 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:12.061755 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20eaed92-2c7a-4490-bc40-4efcb335b0bb-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj\" (UID: \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:06:12.061935 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:12.061918 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20eaed92-2c7a-4490-bc40-4efcb335b0bb-maas-api-tls\") pod \"maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj\" (UID: \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:06:12.062433 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:12.062366 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20eaed92-2c7a-4490-bc40-4efcb335b0bb-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj\" (UID: \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:06:12.064333 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:12.064310 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20eaed92-2c7a-4490-bc40-4efcb335b0bb-metrics-certs\") pod \"maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj\" (UID: \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:06:12.064551 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:12.064528 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20eaed92-2c7a-4490-bc40-4efcb335b0bb-maas-api-tls\") pod \"maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj\" (UID: \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:06:12.085844 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:12.085792 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcsrf\" (UniqueName: \"kubernetes.io/projected/20eaed92-2c7a-4490-bc40-4efcb335b0bb-kube-api-access-wcsrf\") pod \"maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj\" (UID: \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:06:12.141777 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:12.141756 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:06:12.459208 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:12.459177 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj"] Oct 30 17:06:12.459532 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:06:12.459510 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20eaed92_2c7a_4490_bc40_4efcb335b0bb.slice/crio-3dc1c3ba853454b03762dbcf2ef3f8ae517e85863da87a3a90b0dda44a45292a WatchSource:0}: Error finding container 3dc1c3ba853454b03762dbcf2ef3f8ae517e85863da87a3a90b0dda44a45292a: Status 404 returned error can't find the container with id 3dc1c3ba853454b03762dbcf2ef3f8ae517e85863da87a3a90b0dda44a45292a Oct 30 17:06:12.460978 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:12.460959 4861 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:06:12.913182 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:12.913149 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" event={"ID":"20eaed92-2c7a-4490-bc40-4efcb335b0bb","Type":"ContainerStarted","Data":"3dc1c3ba853454b03762dbcf2ef3f8ae517e85863da87a3a90b0dda44a45292a"} Oct 30 17:06:13.922657 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:13.922619 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" event={"ID":"20eaed92-2c7a-4490-bc40-4efcb335b0bb","Type":"ContainerStarted","Data":"81fcb0c0f8f7fc070a8e9770523cab9a1f5532adb1d5eb73b025ae806a5818e0"} Oct 30 17:06:13.923122 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:13.922863 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:06:13.968592 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:13.968545 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" podStartSLOduration=2.45426087 podStartE2EDuration="2.968534174s" totalImagesPullingTime="514.273304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:13.965490344 +0000 UTC m=+2604.308164397" watchObservedRunningTime="2026-10-30 17:06:13.968534174 +0000 UTC m=+2604.311208212" Oct 30 17:06:19.936352 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:19.936317 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:06:20.290992 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:20.290955 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:06:25.972251 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:25.972217 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl"] Oct 30 17:06:25.977855 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:25.977831 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:25.996994 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:25.996967 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl"] Oct 30 17:06:26.002490 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:26.002460 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl\" (UID: \"f468c78f-4fe6-4d84-a07f-90961ddda85a\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:26.002643 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:26.002527 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snwqn\" (UniqueName: \"kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-kube-api-access-snwqn\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl\" (UID: \"f468c78f-4fe6-4d84-a07f-90961ddda85a\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:26.002756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:26.002698 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f468c78f-4fe6-4d84-a07f-90961ddda85a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl\" (UID: \"f468c78f-4fe6-4d84-a07f-90961ddda85a\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:26.002894 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:26.002876 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f468c78f-4fe6-4d84-a07f-90961ddda85a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl\" (UID: \"f468c78f-4fe6-4d84-a07f-90961ddda85a\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:26.104089 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:26.104055 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f468c78f-4fe6-4d84-a07f-90961ddda85a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl\" (UID: \"f468c78f-4fe6-4d84-a07f-90961ddda85a\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:26.104283 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:26.104156 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f468c78f-4fe6-4d84-a07f-90961ddda85a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl\" (UID: \"f468c78f-4fe6-4d84-a07f-90961ddda85a\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:26.104283 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:26.104214 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl\" (UID: \"f468c78f-4fe6-4d84-a07f-90961ddda85a\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:26.104283 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:26.104258 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snwqn\" (UniqueName: \"kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-kube-api-access-snwqn\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl\" (UID: \"f468c78f-4fe6-4d84-a07f-90961ddda85a\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:26.104951 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:26.104924 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f468c78f-4fe6-4d84-a07f-90961ddda85a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl\" (UID: \"f468c78f-4fe6-4d84-a07f-90961ddda85a\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:26.105357 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:26.105329 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f468c78f-4fe6-4d84-a07f-90961ddda85a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl\" (UID: \"f468c78f-4fe6-4d84-a07f-90961ddda85a\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:26.107205 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:26.107185 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl\" (UID: \"f468c78f-4fe6-4d84-a07f-90961ddda85a\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:26.172385 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:26.172337 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snwqn\" (UniqueName: \"kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-kube-api-access-snwqn\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl\" (UID: \"f468c78f-4fe6-4d84-a07f-90961ddda85a\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:26.296082 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:26.295978 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:26.610988 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:26.610959 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl"] Oct 30 17:06:26.611945 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:06:26.611914 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf468c78f_4fe6_4d84_a07f_90961ddda85a.slice/crio-245548e75460f5da5daf7a7d11ab2035eb84c749236370cc137559d4fa38ab49 WatchSource:0}: Error finding container 245548e75460f5da5daf7a7d11ab2035eb84c749236370cc137559d4fa38ab49: Status 404 returned error can't find the container with id 245548e75460f5da5daf7a7d11ab2035eb84c749236370cc137559d4fa38ab49 Oct 30 17:06:27.022541 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:27.022506 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" event={"ID":"f468c78f-4fe6-4d84-a07f-90961ddda85a","Type":"ContainerStarted","Data":"d58f75dc8e485ab1b04371dbfd0a8a7d1b377f257df4cf58e81ac398f1d53a95"} Oct 30 17:06:27.023061 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:27.022572 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" event={"ID":"f468c78f-4fe6-4d84-a07f-90961ddda85a","Type":"ContainerStarted","Data":"245548e75460f5da5daf7a7d11ab2035eb84c749236370cc137559d4fa38ab49"} Oct 30 17:06:27.023061 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:27.022604 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:27.066166 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:27.066113 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" podStartSLOduration=2.066100138 podStartE2EDuration="2.066100138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:27.065491987 +0000 UTC m=+2617.408166026" watchObservedRunningTime="2026-10-30 17:06:27.066100138 +0000 UTC m=+2617.408774177" Oct 30 17:06:36.954419 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:36.954385 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-832ade4be86f: secret "agc-cred-832ade4be86f" not found Oct 30 17:06:36.954419 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:36.954411 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx: secret "agc-cred-832ade4be86f" not found Oct 30 17:06:36.955052 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:36.954488 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-provider-credentials podName:9ee4ae9f-daa3-4980-9fd7-57e4aa728157 nodeName:}" failed. No retries permitted until 2026-10-30 17:06:37.454467516 +0000 UTC m=+2627.797141532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" (UID: "9ee4ae9f-daa3-4980-9fd7-57e4aa728157") : secret "agc-cred-832ade4be86f" not found Oct 30 17:06:37.461235 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:37.461204 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-7c68f4c17b3a: secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:37.461235 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:37.461238 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-832ade4be86f: secret "agc-cred-832ade4be86f" not found Oct 30 17:06:37.461453 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:37.461252 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx: [secret "agc-cred-7c68f4c17b3a" not found, secret "agc-cred-832ade4be86f" not found] Oct 30 17:06:37.461453 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:37.461324 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-provider-credentials podName:9ee4ae9f-daa3-4980-9fd7-57e4aa728157 nodeName:}" failed. No retries permitted until 2026-10-30 17:06:38.461304756 +0000 UTC m=+2628.803978787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" (UID: "9ee4ae9f-daa3-4980-9fd7-57e4aa728157") : [secret "agc-cred-7c68f4c17b3a" not found, secret "agc-cred-832ade4be86f" not found] Oct 30 17:06:38.053748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:38.051635 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:38.143485 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:38.143452 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx"] Oct 30 17:06:38.177420 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:38.177371 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-7c68f4c17b3a: secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:38.177660 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:38.177641 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl: secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:38.177910 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:38.177894 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials podName:f468c78f-4fe6-4d84-a07f-90961ddda85a nodeName:}" failed. No retries permitted until 2026-10-30 17:06:38.677873405 +0000 UTC m=+2629.020547444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" (UID: "f468c78f-4fe6-4d84-a07f-90961ddda85a") : secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:38.481508 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:38.481479 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-7c68f4c17b3a: secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:38.481508 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:38.481503 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-832ade4be86f: secret "agc-cred-832ade4be86f" not found Oct 30 17:06:38.481957 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:38.481521 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx: [secret "agc-cred-7c68f4c17b3a" not found, secret "agc-cred-832ade4be86f" not found] Oct 30 17:06:38.481957 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:38.481602 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-provider-credentials podName:9ee4ae9f-daa3-4980-9fd7-57e4aa728157 nodeName:}" failed. No retries permitted until 2026-10-30 17:06:40.481586318 +0000 UTC m=+2630.824260333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" (UID: "9ee4ae9f-daa3-4980-9fd7-57e4aa728157") : [secret "agc-cred-7c68f4c17b3a" not found, secret "agc-cred-832ade4be86f" not found] Oct 30 17:06:38.683967 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:38.683935 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-7c68f4c17b3a: secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:38.683967 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:38.683960 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl: secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:38.684168 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:38.684026 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials podName:f468c78f-4fe6-4d84-a07f-90961ddda85a nodeName:}" failed. No retries permitted until 2026-10-30 17:06:39.684011138 +0000 UTC m=+2630.026685154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" (UID: "f468c78f-4fe6-4d84-a07f-90961ddda85a") : secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:39.555662 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.555622 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj"] Oct 30 17:06:39.561768 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.561716 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:06:39.578667 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.578641 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"kube-root-ca.crt\"" Oct 30 17:06:39.578844 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.578673 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"routing-overlay\"" Oct 30 17:06:39.578844 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.578673 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"payload-processing-external-model-plugins-e2e-extmod-a-f94fb5\"" Oct 30 17:06:39.579040 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.578711 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"openshift-service-ca.crt\"" Oct 30 17:06:39.584568 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.584546 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj"] Oct 30 17:06:39.592752 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.592712 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"e2e-cmp-legacy-75844b-provider-cred\"" Oct 30 17:06:39.592868 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.592844 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"payload-processing-external-model-e2e--ac654823-dockercfg-svm8r\"" Oct 30 17:06:39.706747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.702026 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8bee04f-708a-4d98-bf9e-4d9d438b336b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj\" (UID: \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:06:39.706747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.702304 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bee04f-708a-4d98-bf9e-4d9d438b336b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj\" (UID: \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:06:39.706747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.702340 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr6kw\" (UniqueName: \"kubernetes.io/projected/f8bee04f-708a-4d98-bf9e-4d9d438b336b-kube-api-access-cr6kw\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj\" (UID: \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:06:39.706747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.702398 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8bee04f-708a-4d98-bf9e-4d9d438b336b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj\" (UID: \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:06:39.706747 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:39.702939 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-7c68f4c17b3a: secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:39.706747 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:39.702959 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl: secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:39.706747 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:39.703040 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials podName:f468c78f-4fe6-4d84-a07f-90961ddda85a nodeName:}" failed. No retries permitted until 2026-10-30 17:06:41.703011642 +0000 UTC m=+2632.045685677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" (UID: "f468c78f-4fe6-4d84-a07f-90961ddda85a") : secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:39.803472 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.803438 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bee04f-708a-4d98-bf9e-4d9d438b336b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj\" (UID: \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:06:39.803472 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.803475 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr6kw\" (UniqueName: \"kubernetes.io/projected/f8bee04f-708a-4d98-bf9e-4d9d438b336b-kube-api-access-cr6kw\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj\" (UID: \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:06:39.803861 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.803511 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8bee04f-708a-4d98-bf9e-4d9d438b336b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj\" (UID: \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:06:39.803861 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.803566 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8bee04f-708a-4d98-bf9e-4d9d438b336b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj\" (UID: \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:06:39.804371 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.804340 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8bee04f-708a-4d98-bf9e-4d9d438b336b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj\" (UID: \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:06:39.804485 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.804383 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bee04f-708a-4d98-bf9e-4d9d438b336b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj\" (UID: \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:06:39.806426 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.806369 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8bee04f-708a-4d98-bf9e-4d9d438b336b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj\" (UID: \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:06:39.828568 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.828540 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr6kw\" (UniqueName: \"kubernetes.io/projected/f8bee04f-708a-4d98-bf9e-4d9d438b336b-kube-api-access-cr6kw\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj\" (UID: \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:06:39.877226 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:39.877204 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:06:40.194762 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:06:40.194706 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bee04f_708a_4d98_bf9e_4d9d438b336b.slice/crio-e2c946a8703d54a1a24794694bc596d0b91faf0daa14126e8a7fa7cd423bf47b WatchSource:0}: Error finding container e2c946a8703d54a1a24794694bc596d0b91faf0daa14126e8a7fa7cd423bf47b: Status 404 returned error can't find the container with id e2c946a8703d54a1a24794694bc596d0b91faf0daa14126e8a7fa7cd423bf47b Oct 30 17:06:40.195049 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:40.195016 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj"] Oct 30 17:06:40.514259 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:40.514229 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-7c68f4c17b3a: secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:40.514259 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:40.514256 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-832ade4be86f: secret "agc-cred-832ade4be86f" not found Oct 30 17:06:40.514466 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:40.514269 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx: [secret "agc-cred-7c68f4c17b3a" not found, secret "agc-cred-832ade4be86f" not found] Oct 30 17:06:40.514466 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:40.514322 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-provider-credentials podName:9ee4ae9f-daa3-4980-9fd7-57e4aa728157 nodeName:}" failed. No retries permitted until 2026-10-30 17:06:44.514307609 +0000 UTC m=+2634.856981624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" (UID: "9ee4ae9f-daa3-4980-9fd7-57e4aa728157") : [secret "agc-cred-7c68f4c17b3a" not found, secret "agc-cred-832ade4be86f" not found] Oct 30 17:06:41.130021 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:41.129969 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" event={"ID":"f8bee04f-708a-4d98-bf9e-4d9d438b336b","Type":"ContainerStarted","Data":"2e402a11ee3aeb4f2793720ef0a2feeca91b9c129e839421cf4e956d47fa20a5"} Oct 30 17:06:41.130465 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:41.130076 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" event={"ID":"f8bee04f-708a-4d98-bf9e-4d9d438b336b","Type":"ContainerStarted","Data":"e2c946a8703d54a1a24794694bc596d0b91faf0daa14126e8a7fa7cd423bf47b"} Oct 30 17:06:41.130465 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:41.130112 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:06:41.177249 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:41.177201 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" podStartSLOduration=2.177187744 podStartE2EDuration="2.177187744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:41.176109089 +0000 UTC m=+2631.518783128" watchObservedRunningTime="2026-10-30 17:06:41.177187744 +0000 UTC m=+2631.519861783" Oct 30 17:06:41.733475 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:41.733439 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-7c68f4c17b3a: secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:41.733475 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:41.733467 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl: secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:41.733799 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:41.733514 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials podName:f468c78f-4fe6-4d84-a07f-90961ddda85a nodeName:}" failed. No retries permitted until 2026-10-30 17:06:45.733501156 +0000 UTC m=+2636.076175172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" (UID: "f468c78f-4fe6-4d84-a07f-90961ddda85a") : secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:43.144423 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:43.144386 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" podUID="9ee4ae9f-daa3-4980-9fd7-57e4aa728157" containerName="payload-processing" containerID="cri-o://885693b8a64c939dad2194c9185e0fe00eb561302c7313ed21f18b9ddadc1d60" gracePeriod=25 Oct 30 17:06:43.449766 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:43.449735 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:06:43.559703 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:43.559673 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-provider-credentials\") pod \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\" (UID: \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\") " Oct 30 17:06:43.559906 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:43.559718 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-plugins-config-volume\") pod \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\" (UID: \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\") " Oct 30 17:06:43.559987 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:43.559914 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-kube-api-access-gmstq\" (UniqueName: \"kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-kube-api-access-gmstq\") pod \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\" (UID: \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\") " Oct 30 17:06:43.559987 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:43.559970 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-routing-overlay-volume\") pod \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\" (UID: \"9ee4ae9f-daa3-4980-9fd7-57e4aa728157\") " Oct 30 17:06:43.560099 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:43.560079 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-plugins-config-volume" pod "9ee4ae9f-daa3-4980-9fd7-57e4aa728157" (UID: "9ee4ae9f-daa3-4980-9fd7-57e4aa728157"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:43.560323 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:43.560302 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-routing-overlay-volume" pod "9ee4ae9f-daa3-4980-9fd7-57e4aa728157" (UID: "9ee4ae9f-daa3-4980-9fd7-57e4aa728157"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:43.560525 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:43.560510 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:06:43.560570 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:43.560528 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:06:43.562150 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:43.562128 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-kube-api-access-gmstq" pod "9ee4ae9f-daa3-4980-9fd7-57e4aa728157" (UID: "9ee4ae9f-daa3-4980-9fd7-57e4aa728157"). InnerVolumeSpecName "kube-api-access-gmstq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:43.562224 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:43.562181 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-provider-credentials" pod "9ee4ae9f-daa3-4980-9fd7-57e4aa728157" (UID: "9ee4ae9f-daa3-4980-9fd7-57e4aa728157"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:43.661635 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:43.661603 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmstq\" (UniqueName: \"kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-kube-api-access-gmstq\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:06:43.661635 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:43.661630 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ee4ae9f-daa3-4980-9fd7-57e4aa728157-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:06:44.180785 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:44.180718 4861 generic.go:356] "Generic (PLEG): container finished" podID="9ee4ae9f-daa3-4980-9fd7-57e4aa728157" containerID="885693b8a64c939dad2194c9185e0fe00eb561302c7313ed21f18b9ddadc1d60" exitCode=0 Oct 30 17:06:44.180785 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:44.180796 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" Oct 30 17:06:44.181283 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:44.180835 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" event={"ID":"9ee4ae9f-daa3-4980-9fd7-57e4aa728157","Type":"ContainerDied","Data":"885693b8a64c939dad2194c9185e0fe00eb561302c7313ed21f18b9ddadc1d60"} Oct 30 17:06:44.181283 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:44.180887 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx" event={"ID":"9ee4ae9f-daa3-4980-9fd7-57e4aa728157","Type":"ContainerDied","Data":"22d7d93a0f76fb342f25bab703f276e403823e47527e12393403edd957816aaa"} Oct 30 17:06:44.181283 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:44.180918 4861 scope.go:122] "RemoveContainer" containerID="885693b8a64c939dad2194c9185e0fe00eb561302c7313ed21f18b9ddadc1d60" Oct 30 17:06:44.191450 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:44.191414 4861 scope.go:122] "RemoveContainer" containerID="885693b8a64c939dad2194c9185e0fe00eb561302c7313ed21f18b9ddadc1d60" Oct 30 17:06:44.191714 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:44.191691 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885693b8a64c939dad2194c9185e0fe00eb561302c7313ed21f18b9ddadc1d60\": container with ID starting with 885693b8a64c939dad2194c9185e0fe00eb561302c7313ed21f18b9ddadc1d60 not found: ID does not exist" containerID="885693b8a64c939dad2194c9185e0fe00eb561302c7313ed21f18b9ddadc1d60" Oct 30 17:06:44.191828 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:44.191740 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885693b8a64c939dad2194c9185e0fe00eb561302c7313ed21f18b9ddadc1d60"} err="failed to get container status \"885693b8a64c939dad2194c9185e0fe00eb561302c7313ed21f18b9ddadc1d60\": rpc error: code = NotFound desc = could not find container \"885693b8a64c939dad2194c9185e0fe00eb561302c7313ed21f18b9ddadc1d60\": container with ID starting with 885693b8a64c939dad2194c9185e0fe00eb561302c7313ed21f18b9ddadc1d60 not found: ID does not exist" Oct 30 17:06:44.239331 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:44.239304 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx"] Oct 30 17:06:44.252790 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:44.252758 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-cfbdrftwx"] Oct 30 17:06:45.787686 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:45.787655 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-7c68f4c17b3a: secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:45.787686 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:45.787683 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl: secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:45.788247 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:45.787765 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials podName:f468c78f-4fe6-4d84-a07f-90961ddda85a nodeName:}" failed. No retries permitted until 2026-10-30 17:06:53.787747698 +0000 UTC m=+2644.130421734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" (UID: "f468c78f-4fe6-4d84-a07f-90961ddda85a") : secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:45.808852 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:45.808817 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ee4ae9f-daa3-4980-9fd7-57e4aa728157" path="/var/lib/kubelet/pods/9ee4ae9f-daa3-4980-9fd7-57e4aa728157/volumes" Oct 30 17:06:49.355538 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:49.355506 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl"] Oct 30 17:06:50.291226 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:50.291189 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:06:52.174235 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:52.174202 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:06:53.795841 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:53.795814 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-3e1593902499: secret "agc-cred-3e1593902499" not found Oct 30 17:06:53.795841 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:53.795837 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-7c68f4c17b3a: secret "agc-cred-7c68f4c17b3a" not found Oct 30 17:06:53.795841 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:53.795847 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl: [secret "agc-cred-3e1593902499" not found, secret "agc-cred-7c68f4c17b3a" not found] Oct 30 17:06:53.796304 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:53.795899 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials podName:f468c78f-4fe6-4d84-a07f-90961ddda85a nodeName:}" failed. No retries permitted until 2026-10-30 17:07:09.795884477 +0000 UTC m=+2660.138558493 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" (UID: "f468c78f-4fe6-4d84-a07f-90961ddda85a") : [secret "agc-cred-3e1593902499" not found, secret "agc-cred-7c68f4c17b3a" not found] Oct 30 17:06:54.356516 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:54.356476 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" podUID="f468c78f-4fe6-4d84-a07f-90961ddda85a" containerName="payload-processing" containerID="cri-o://d58f75dc8e485ab1b04371dbfd0a8a7d1b377f257df4cf58e81ac398f1d53a95" gracePeriod=25 Oct 30 17:06:54.666496 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:54.666467 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:54.825466 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:54.825433 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-kube-api-access-snwqn\" (UniqueName: \"kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-kube-api-access-snwqn\") pod \"f468c78f-4fe6-4d84-a07f-90961ddda85a\" (UID: \"f468c78f-4fe6-4d84-a07f-90961ddda85a\") " Oct 30 17:06:54.825875 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:54.825569 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f468c78f-4fe6-4d84-a07f-90961ddda85a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f468c78f-4fe6-4d84-a07f-90961ddda85a-routing-overlay-volume\") pod \"f468c78f-4fe6-4d84-a07f-90961ddda85a\" (UID: \"f468c78f-4fe6-4d84-a07f-90961ddda85a\") " Oct 30 17:06:54.825875 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:54.825643 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials\") pod \"f468c78f-4fe6-4d84-a07f-90961ddda85a\" (UID: \"f468c78f-4fe6-4d84-a07f-90961ddda85a\") " Oct 30 17:06:54.825875 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:54.825793 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f468c78f-4fe6-4d84-a07f-90961ddda85a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f468c78f-4fe6-4d84-a07f-90961ddda85a-plugins-config-volume\") pod \"f468c78f-4fe6-4d84-a07f-90961ddda85a\" (UID: \"f468c78f-4fe6-4d84-a07f-90961ddda85a\") " Oct 30 17:06:54.826013 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:54.825979 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f468c78f-4fe6-4d84-a07f-90961ddda85a-routing-overlay-volume" pod "f468c78f-4fe6-4d84-a07f-90961ddda85a" (UID: "f468c78f-4fe6-4d84-a07f-90961ddda85a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:54.826165 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:54.826145 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f468c78f-4fe6-4d84-a07f-90961ddda85a-plugins-config-volume" pod "f468c78f-4fe6-4d84-a07f-90961ddda85a" (UID: "f468c78f-4fe6-4d84-a07f-90961ddda85a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:06:54.826408 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:54.826395 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f468c78f-4fe6-4d84-a07f-90961ddda85a-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:06:54.826442 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:54.826411 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f468c78f-4fe6-4d84-a07f-90961ddda85a-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:06:54.827944 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:54.827916 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-kube-api-access-snwqn" pod "f468c78f-4fe6-4d84-a07f-90961ddda85a" (UID: "f468c78f-4fe6-4d84-a07f-90961ddda85a"). InnerVolumeSpecName "kube-api-access-snwqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:54.828419 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:54.828402 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials" pod "f468c78f-4fe6-4d84-a07f-90961ddda85a" (UID: "f468c78f-4fe6-4d84-a07f-90961ddda85a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:54.927823 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:54.927713 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:06:54.927823 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:54.927768 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snwqn\" (UniqueName: \"kubernetes.io/projected/f468c78f-4fe6-4d84-a07f-90961ddda85a-kube-api-access-snwqn\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:06:55.280614 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:55.280574 4861 generic.go:356] "Generic (PLEG): container finished" podID="f468c78f-4fe6-4d84-a07f-90961ddda85a" containerID="d58f75dc8e485ab1b04371dbfd0a8a7d1b377f257df4cf58e81ac398f1d53a95" exitCode=0 Oct 30 17:06:55.280823 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:55.280633 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" Oct 30 17:06:55.280823 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:55.280658 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" event={"ID":"f468c78f-4fe6-4d84-a07f-90961ddda85a","Type":"ContainerDied","Data":"d58f75dc8e485ab1b04371dbfd0a8a7d1b377f257df4cf58e81ac398f1d53a95"} Oct 30 17:06:55.280823 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:55.280717 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl" event={"ID":"f468c78f-4fe6-4d84-a07f-90961ddda85a","Type":"ContainerDied","Data":"245548e75460f5da5daf7a7d11ab2035eb84c749236370cc137559d4fa38ab49"} Oct 30 17:06:55.280823 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:55.280779 4861 scope.go:122] "RemoveContainer" containerID="d58f75dc8e485ab1b04371dbfd0a8a7d1b377f257df4cf58e81ac398f1d53a95" Oct 30 17:06:55.295381 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:55.295360 4861 scope.go:122] "RemoveContainer" containerID="d58f75dc8e485ab1b04371dbfd0a8a7d1b377f257df4cf58e81ac398f1d53a95" Oct 30 17:06:55.295639 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:06:55.295618 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58f75dc8e485ab1b04371dbfd0a8a7d1b377f257df4cf58e81ac398f1d53a95\": container with ID starting with d58f75dc8e485ab1b04371dbfd0a8a7d1b377f257df4cf58e81ac398f1d53a95 not found: ID does not exist" containerID="d58f75dc8e485ab1b04371dbfd0a8a7d1b377f257df4cf58e81ac398f1d53a95" Oct 30 17:06:55.295698 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:55.295647 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58f75dc8e485ab1b04371dbfd0a8a7d1b377f257df4cf58e81ac398f1d53a95"} err="failed to get container status \"d58f75dc8e485ab1b04371dbfd0a8a7d1b377f257df4cf58e81ac398f1d53a95\": rpc error: code = NotFound desc = could not find container \"d58f75dc8e485ab1b04371dbfd0a8a7d1b377f257df4cf58e81ac398f1d53a95\": container with ID starting with d58f75dc8e485ab1b04371dbfd0a8a7d1b377f257df4cf58e81ac398f1d53a95 not found: ID does not exist" Oct 30 17:06:55.339531 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:55.339485 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl"] Oct 30 17:06:55.352218 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:55.352187 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-76bd5k6xl"] Oct 30 17:06:55.809487 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:06:55.809457 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f468c78f-4fe6-4d84-a07f-90961ddda85a" path="/var/lib/kubelet/pods/f468c78f-4fe6-4d84-a07f-90961ddda85a/volumes" Oct 30 17:07:08.852689 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.852479 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw"] Oct 30 17:07:08.854604 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.854573 4861 cpu_manager.go:414] "containerMap: removing container" podUID="9ee4ae9f-daa3-4980-9fd7-57e4aa728157" containerName="payload-processing" Oct 30 17:07:08.860802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.860773 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee4ae9f-daa3-4980-9fd7-57e4aa728157" containerName="payload-processing" Oct 30 17:07:08.861014 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.861002 4861 cpu_manager.go:414] "containerMap: removing container" podUID="f468c78f-4fe6-4d84-a07f-90961ddda85a" containerName="payload-processing" Oct 30 17:07:08.861112 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.861101 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f468c78f-4fe6-4d84-a07f-90961ddda85a" containerName="payload-processing" Oct 30 17:07:08.862150 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.862129 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="f468c78f-4fe6-4d84-a07f-90961ddda85a" containerName="payload-processing" Oct 30 17:07:08.862324 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.862311 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee4ae9f-daa3-4980-9fd7-57e4aa728157" containerName="payload-processing" Oct 30 17:07:08.874555 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.874488 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw"] Oct 30 17:07:08.874742 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.874602 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:08.907168 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.907138 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"e2e-cfl-9d7fa9-provider-cred\"" Oct 30 17:07:08.907326 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.907195 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"agc-cred-c1ddc76883fe\"" Oct 30 17:07:08.907326 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.907203 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"routing-overlay\"" Oct 30 17:07:08.907326 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.907142 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"payload-processing-external-model-e2e--8e3f1132-dockercfg-gjbn7\"" Oct 30 17:07:08.907326 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.907181 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"kube-root-ca.crt\"" Oct 30 17:07:08.907326 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.907294 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"openshift-service-ca.crt\"" Oct 30 17:07:08.907326 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.907299 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2589b5\"" Oct 30 17:07:08.946341 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.946312 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:08.946484 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.946355 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45234eff-8d0e-4c94-8af1-87b2d11b5c81-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:08.946538 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.946510 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45234eff-8d0e-4c94-8af1-87b2d11b5c81-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:08.946606 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:08.946590 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sf5d\" (UniqueName: \"kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-kube-api-access-9sf5d\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:09.048156 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.048125 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:09.048156 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.048155 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45234eff-8d0e-4c94-8af1-87b2d11b5c81-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:09.048384 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.048195 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45234eff-8d0e-4c94-8af1-87b2d11b5c81-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:09.048384 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.048230 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sf5d\" (UniqueName: \"kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-kube-api-access-9sf5d\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:09.048384 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:09.048247 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2589b5/agc-cred-c1ddc76883fe: secret "agc-cred-c1ddc76883fe" not found Oct 30 17:07:09.048384 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:09.048267 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw: secret "agc-cred-c1ddc76883fe" not found Oct 30 17:07:09.048384 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:09.048320 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials podName:45234eff-8d0e-4c94-8af1-87b2d11b5c81 nodeName:}" failed. No retries permitted until 2026-10-30 17:07:09.548305222 +0000 UTC m=+2659.890979241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" (UID: "45234eff-8d0e-4c94-8af1-87b2d11b5c81") : secret "agc-cred-c1ddc76883fe" not found Oct 30 17:07:09.048851 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.048826 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45234eff-8d0e-4c94-8af1-87b2d11b5c81-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:09.049140 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.049117 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45234eff-8d0e-4c94-8af1-87b2d11b5c81-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:09.074403 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.074381 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sf5d\" (UniqueName: \"kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-kube-api-access-9sf5d\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:09.553943 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.553909 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:09.554122 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:09.554058 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2589b5/agc-cred-c1ddc76883fe: secret "agc-cred-c1ddc76883fe" not found Oct 30 17:07:09.554122 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:09.554082 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw: secret "agc-cred-c1ddc76883fe" not found Oct 30 17:07:09.554208 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:09.554142 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials podName:45234eff-8d0e-4c94-8af1-87b2d11b5c81 nodeName:}" failed. No retries permitted until 2026-10-30 17:07:10.554127945 +0000 UTC m=+2660.896801961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" (UID: "45234eff-8d0e-4c94-8af1-87b2d11b5c81") : secret "agc-cred-c1ddc76883fe" not found Oct 30 17:07:09.688515 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.688481 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm"] Oct 30 17:07:09.693461 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.693443 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:07:09.715919 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.715892 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm"] Oct 30 17:07:09.756561 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.756532 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm\" (UID: \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:07:09.756701 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.756592 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm\" (UID: \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:07:09.756701 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.756687 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm\" (UID: \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:07:09.756854 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.756841 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49ptg\" (UniqueName: \"kubernetes.io/projected/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-kube-api-access-49ptg\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm\" (UID: \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:07:09.857992 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.857895 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm\" (UID: \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:07:09.858502 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.858038 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49ptg\" (UniqueName: \"kubernetes.io/projected/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-kube-api-access-49ptg\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm\" (UID: \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:07:09.858502 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.858161 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm\" (UID: \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:07:09.858502 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.858235 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm\" (UID: \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:07:09.858783 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.858678 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm\" (UID: \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:07:09.858902 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.858880 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm\" (UID: \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:07:09.860935 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.860913 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm\" (UID: \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:07:09.883111 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:09.883081 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49ptg\" (UniqueName: \"kubernetes.io/projected/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-kube-api-access-49ptg\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm\" (UID: \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:07:10.009471 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:10.009439 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:07:10.321625 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:10.321590 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm"] Oct 30 17:07:10.322163 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:07:10.322136 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3468a16e_dad6_45b2_b516_dc8bf6c86ebd.slice/crio-a1195756f82f47d4eb244788d5b985d40b8cbd9356b05f07cb23df7870f76ee9 WatchSource:0}: Error finding container a1195756f82f47d4eb244788d5b985d40b8cbd9356b05f07cb23df7870f76ee9: Status 404 returned error can't find the container with id a1195756f82f47d4eb244788d5b985d40b8cbd9356b05f07cb23df7870f76ee9 Oct 30 17:07:10.441567 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:10.441529 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" event={"ID":"3468a16e-dad6-45b2-b516-dc8bf6c86ebd","Type":"ContainerStarted","Data":"9e170f553ae71d4daab3661f9a49d302f8f23d4c8a59a831d640944fa1aeffd9"} Oct 30 17:07:10.441700 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:10.441583 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" event={"ID":"3468a16e-dad6-45b2-b516-dc8bf6c86ebd","Type":"ContainerStarted","Data":"a1195756f82f47d4eb244788d5b985d40b8cbd9356b05f07cb23df7870f76ee9"} Oct 30 17:07:10.441700 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:10.441614 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:07:10.486888 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:10.486844 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" podStartSLOduration=1.486829274 podStartE2EDuration="1.486829274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:10.485965878 +0000 UTC m=+2660.828639910" watchObservedRunningTime="2026-10-30 17:07:10.486829274 +0000 UTC m=+2660.829503311" Oct 30 17:07:10.568987 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:10.568909 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:10.569153 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:10.569065 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2589b5/agc-cred-c1ddc76883fe: secret "agc-cred-c1ddc76883fe" not found Oct 30 17:07:10.569153 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:10.569092 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw: secret "agc-cred-c1ddc76883fe" not found Oct 30 17:07:10.569153 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:10.569149 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials podName:45234eff-8d0e-4c94-8af1-87b2d11b5c81 nodeName:}" failed. No retries permitted until 2026-10-30 17:07:12.569134054 +0000 UTC m=+2662.911808070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" (UID: "45234eff-8d0e-4c94-8af1-87b2d11b5c81") : secret "agc-cred-c1ddc76883fe" not found Oct 30 17:07:12.602747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:12.601898 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:12.603308 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:12.603013 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2589b5/agc-cred-c1ddc76883fe: secret "agc-cred-c1ddc76883fe" not found Oct 30 17:07:12.603308 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:12.603052 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw: secret "agc-cred-c1ddc76883fe" not found Oct 30 17:07:12.603308 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:12.603113 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials podName:45234eff-8d0e-4c94-8af1-87b2d11b5c81 nodeName:}" failed. No retries permitted until 2026-10-30 17:07:16.603093609 +0000 UTC m=+2666.945767637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" (UID: "45234eff-8d0e-4c94-8af1-87b2d11b5c81") : secret "agc-cred-c1ddc76883fe" not found Oct 30 17:07:16.692888 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:16.692848 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:16.693583 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:16.692994 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2589b5/agc-cred-c1ddc76883fe: secret "agc-cred-c1ddc76883fe" not found Oct 30 17:07:16.693583 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:16.693020 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw: secret "agc-cred-c1ddc76883fe" not found Oct 30 17:07:16.693583 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:16.693094 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials podName:45234eff-8d0e-4c94-8af1-87b2d11b5c81 nodeName:}" failed. No retries permitted until 2026-10-30 17:07:24.693071087 +0000 UTC m=+2675.035745122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" (UID: "45234eff-8d0e-4c94-8af1-87b2d11b5c81") : secret "agc-cred-c1ddc76883fe" not found Oct 30 17:07:18.538812 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.538780 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8"] Oct 30 17:07:18.546822 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.544938 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:07:18.564124 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.564072 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8"] Oct 30 17:07:18.573127 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.573093 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"agc-cred-2d7c56b1c50f\"" Oct 30 17:07:18.573127 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.573101 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3106d6\"/\"agc-cred-129cce3672c2\"" Oct 30 17:07:18.619364 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.619331 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a34df277-cd10-4cc9-ad56-5ebf552938f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8\" (UID: \"a34df277-cd10-4cc9-ad56-5ebf552938f5\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:07:18.619527 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.619513 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34df277-cd10-4cc9-ad56-5ebf552938f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8\" (UID: \"a34df277-cd10-4cc9-ad56-5ebf552938f5\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:07:18.619662 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.619639 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc5rq\" (UniqueName: \"kubernetes.io/projected/a34df277-cd10-4cc9-ad56-5ebf552938f5-kube-api-access-zc5rq\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8\" (UID: \"a34df277-cd10-4cc9-ad56-5ebf552938f5\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:07:18.619711 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.619676 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a34df277-cd10-4cc9-ad56-5ebf552938f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8\" (UID: \"a34df277-cd10-4cc9-ad56-5ebf552938f5\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:07:18.720761 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.720701 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a34df277-cd10-4cc9-ad56-5ebf552938f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8\" (UID: \"a34df277-cd10-4cc9-ad56-5ebf552938f5\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:07:18.720936 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.720878 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34df277-cd10-4cc9-ad56-5ebf552938f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8\" (UID: \"a34df277-cd10-4cc9-ad56-5ebf552938f5\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:07:18.721000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.720988 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc5rq\" (UniqueName: \"kubernetes.io/projected/a34df277-cd10-4cc9-ad56-5ebf552938f5-kube-api-access-zc5rq\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8\" (UID: \"a34df277-cd10-4cc9-ad56-5ebf552938f5\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:07:18.721064 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.721024 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a34df277-cd10-4cc9-ad56-5ebf552938f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8\" (UID: \"a34df277-cd10-4cc9-ad56-5ebf552938f5\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:07:18.721859 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.721836 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34df277-cd10-4cc9-ad56-5ebf552938f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8\" (UID: \"a34df277-cd10-4cc9-ad56-5ebf552938f5\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:07:18.722000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.721976 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a34df277-cd10-4cc9-ad56-5ebf552938f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8\" (UID: \"a34df277-cd10-4cc9-ad56-5ebf552938f5\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:07:18.724204 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.724180 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a34df277-cd10-4cc9-ad56-5ebf552938f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8\" (UID: \"a34df277-cd10-4cc9-ad56-5ebf552938f5\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:07:18.746127 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.746096 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc5rq\" (UniqueName: \"kubernetes.io/projected/a34df277-cd10-4cc9-ad56-5ebf552938f5-kube-api-access-zc5rq\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8\" (UID: \"a34df277-cd10-4cc9-ad56-5ebf552938f5\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:07:18.865774 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:18.865701 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:07:19.181629 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:19.181601 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8"] Oct 30 17:07:19.182254 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:07:19.182205 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34df277_cd10_4cc9_ad56_5ebf552938f5.slice/crio-11c0ca2441a15503dd35ec47e06b2f5da6ea77215751db314e3f145061e31547 WatchSource:0}: Error finding container 11c0ca2441a15503dd35ec47e06b2f5da6ea77215751db314e3f145061e31547: Status 404 returned error can't find the container with id 11c0ca2441a15503dd35ec47e06b2f5da6ea77215751db314e3f145061e31547 Oct 30 17:07:19.571864 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:19.571823 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" event={"ID":"a34df277-cd10-4cc9-ad56-5ebf552938f5","Type":"ContainerStarted","Data":"a6e38680a0757acb253609e3274556992191cbe59cf1276c63e77d9a3abe6ab4"} Oct 30 17:07:19.572280 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:19.571890 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" event={"ID":"a34df277-cd10-4cc9-ad56-5ebf552938f5","Type":"ContainerStarted","Data":"11c0ca2441a15503dd35ec47e06b2f5da6ea77215751db314e3f145061e31547"} Oct 30 17:07:19.572280 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:19.571918 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:07:19.620308 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:19.620251 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" podStartSLOduration=1.6202347480000001 podStartE2EDuration="1.620234748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:19.616336099 +0000 UTC m=+2669.959010155" watchObservedRunningTime="2026-10-30 17:07:19.620234748 +0000 UTC m=+2669.962908786" Oct 30 17:07:20.291038 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:20.290997 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:07:21.460116 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:21.460082 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:07:21.544377 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:21.544344 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw"] Oct 30 17:07:21.544764 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:21.544706 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" podUID="45234eff-8d0e-4c94-8af1-87b2d11b5c81" Oct 30 17:07:21.586769 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:21.586739 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:21.596606 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:21.596579 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:21.762423 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:21.762392 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45234eff-8d0e-4c94-8af1-87b2d11b5c81-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45234eff-8d0e-4c94-8af1-87b2d11b5c81-routing-overlay-volume\") pod \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " Oct 30 17:07:21.762602 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:21.762561 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-kube-api-access-9sf5d\" (UniqueName: \"kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-kube-api-access-9sf5d\") pod \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " Oct 30 17:07:21.762602 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:21.762593 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45234eff-8d0e-4c94-8af1-87b2d11b5c81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45234eff-8d0e-4c94-8af1-87b2d11b5c81-plugins-config-volume\") pod \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\" (UID: \"45234eff-8d0e-4c94-8af1-87b2d11b5c81\") " Oct 30 17:07:21.762784 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:21.762762 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45234eff-8d0e-4c94-8af1-87b2d11b5c81-routing-overlay-volume" pod "45234eff-8d0e-4c94-8af1-87b2d11b5c81" (UID: "45234eff-8d0e-4c94-8af1-87b2d11b5c81"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:21.763006 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:21.762986 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45234eff-8d0e-4c94-8af1-87b2d11b5c81-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:07:21.763173 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:21.763062 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45234eff-8d0e-4c94-8af1-87b2d11b5c81-plugins-config-volume" pod "45234eff-8d0e-4c94-8af1-87b2d11b5c81" (UID: "45234eff-8d0e-4c94-8af1-87b2d11b5c81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:21.765031 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:21.765009 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-kube-api-access-9sf5d" pod "45234eff-8d0e-4c94-8af1-87b2d11b5c81" (UID: "45234eff-8d0e-4c94-8af1-87b2d11b5c81"). InnerVolumeSpecName "kube-api-access-9sf5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:21.864659 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:21.864622 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45234eff-8d0e-4c94-8af1-87b2d11b5c81-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:07:21.864659 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:21.864652 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sf5d\" (UniqueName: \"kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-kube-api-access-9sf5d\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:07:22.593675 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:22.593642 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw" Oct 30 17:07:22.686228 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:22.686194 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw"] Oct 30 17:07:22.692018 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:22.691988 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-84b8txbzw"] Oct 30 17:07:22.788666 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:22.788633 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45234eff-8d0e-4c94-8af1-87b2d11b5c81-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:07:23.809448 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:23.809412 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45234eff-8d0e-4c94-8af1-87b2d11b5c81" path="/var/lib/kubelet/pods/45234eff-8d0e-4c94-8af1-87b2d11b5c81/volumes" Oct 30 17:07:30.582817 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:30.582790 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:07:30.666301 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:30.666265 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m"] Oct 30 17:07:35.666792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:35.666754 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" podUID="aeb6353a-f1de-4669-afd7-a17f36d2a194" containerName="payload-processing" containerID="cri-o://52cd4df6b164ec73cf9e89d3c32e2dfd96f90d4bc3feaa4cc544492617024f0e" gracePeriod=25 Oct 30 17:07:35.977693 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:35.977665 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:07:36.070236 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.070206 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeb6353a-f1de-4669-afd7-a17f36d2a194-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb6353a-f1de-4669-afd7-a17f36d2a194-plugins-config-volume\") pod \"aeb6353a-f1de-4669-afd7-a17f36d2a194\" (UID: \"aeb6353a-f1de-4669-afd7-a17f36d2a194\") " Oct 30 17:07:36.070391 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.070347 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeb6353a-f1de-4669-afd7-a17f36d2a194-provider-credentials\" (UniqueName: \"kubernetes.io/projected/aeb6353a-f1de-4669-afd7-a17f36d2a194-provider-credentials\") pod \"aeb6353a-f1de-4669-afd7-a17f36d2a194\" (UID: \"aeb6353a-f1de-4669-afd7-a17f36d2a194\") " Oct 30 17:07:36.070472 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.070454 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aeb6353a-f1de-4669-afd7-a17f36d2a194-kube-api-access-m8w92\" (UniqueName: \"kubernetes.io/projected/aeb6353a-f1de-4669-afd7-a17f36d2a194-kube-api-access-m8w92\") pod \"aeb6353a-f1de-4669-afd7-a17f36d2a194\" (UID: \"aeb6353a-f1de-4669-afd7-a17f36d2a194\") " Oct 30 17:07:36.070516 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.070495 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aeb6353a-f1de-4669-afd7-a17f36d2a194-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aeb6353a-f1de-4669-afd7-a17f36d2a194-routing-overlay-volume\") pod \"aeb6353a-f1de-4669-afd7-a17f36d2a194\" (UID: \"aeb6353a-f1de-4669-afd7-a17f36d2a194\") " Oct 30 17:07:36.070671 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.070642 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb6353a-f1de-4669-afd7-a17f36d2a194-plugins-config-volume" pod "aeb6353a-f1de-4669-afd7-a17f36d2a194" (UID: "aeb6353a-f1de-4669-afd7-a17f36d2a194"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:36.071120 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.071094 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb6353a-f1de-4669-afd7-a17f36d2a194-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:07:36.071226 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.071117 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb6353a-f1de-4669-afd7-a17f36d2a194-routing-overlay-volume" pod "aeb6353a-f1de-4669-afd7-a17f36d2a194" (UID: "aeb6353a-f1de-4669-afd7-a17f36d2a194"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:36.072858 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.072825 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb6353a-f1de-4669-afd7-a17f36d2a194-provider-credentials" pod "aeb6353a-f1de-4669-afd7-a17f36d2a194" (UID: "aeb6353a-f1de-4669-afd7-a17f36d2a194"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:36.072975 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.072926 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb6353a-f1de-4669-afd7-a17f36d2a194-kube-api-access-m8w92" pod "aeb6353a-f1de-4669-afd7-a17f36d2a194" (UID: "aeb6353a-f1de-4669-afd7-a17f36d2a194"). InnerVolumeSpecName "kube-api-access-m8w92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:36.172393 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.172361 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aeb6353a-f1de-4669-afd7-a17f36d2a194-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:07:36.172393 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.172388 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8w92\" (UniqueName: \"kubernetes.io/projected/aeb6353a-f1de-4669-afd7-a17f36d2a194-kube-api-access-m8w92\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:07:36.172393 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.172397 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aeb6353a-f1de-4669-afd7-a17f36d2a194-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:07:36.708437 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.708405 4861 generic.go:356] "Generic (PLEG): container finished" podID="aeb6353a-f1de-4669-afd7-a17f36d2a194" containerID="52cd4df6b164ec73cf9e89d3c32e2dfd96f90d4bc3feaa4cc544492617024f0e" exitCode=0 Oct 30 17:07:36.708940 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.708453 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" Oct 30 17:07:36.708940 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.708489 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" event={"ID":"aeb6353a-f1de-4669-afd7-a17f36d2a194","Type":"ContainerDied","Data":"52cd4df6b164ec73cf9e89d3c32e2dfd96f90d4bc3feaa4cc544492617024f0e"} Oct 30 17:07:36.708940 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.708553 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m" event={"ID":"aeb6353a-f1de-4669-afd7-a17f36d2a194","Type":"ContainerDied","Data":"2fcff230f78cf71e770fde49fd8d0c45bd16ab8306fac2cc83b2e5bce35ee6c1"} Oct 30 17:07:36.708940 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.708586 4861 scope.go:122] "RemoveContainer" containerID="52cd4df6b164ec73cf9e89d3c32e2dfd96f90d4bc3feaa4cc544492617024f0e" Oct 30 17:07:36.722713 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.722696 4861 scope.go:122] "RemoveContainer" containerID="52cd4df6b164ec73cf9e89d3c32e2dfd96f90d4bc3feaa4cc544492617024f0e" Oct 30 17:07:36.723047 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:36.723013 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52cd4df6b164ec73cf9e89d3c32e2dfd96f90d4bc3feaa4cc544492617024f0e\": container with ID starting with 52cd4df6b164ec73cf9e89d3c32e2dfd96f90d4bc3feaa4cc544492617024f0e not found: ID does not exist" containerID="52cd4df6b164ec73cf9e89d3c32e2dfd96f90d4bc3feaa4cc544492617024f0e" Oct 30 17:07:36.723146 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.723058 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52cd4df6b164ec73cf9e89d3c32e2dfd96f90d4bc3feaa4cc544492617024f0e"} err="failed to get container status \"52cd4df6b164ec73cf9e89d3c32e2dfd96f90d4bc3feaa4cc544492617024f0e\": rpc error: code = NotFound desc = could not find container \"52cd4df6b164ec73cf9e89d3c32e2dfd96f90d4bc3feaa4cc544492617024f0e\": container with ID starting with 52cd4df6b164ec73cf9e89d3c32e2dfd96f90d4bc3feaa4cc544492617024f0e not found: ID does not exist" Oct 30 17:07:36.768057 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.768026 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m"] Oct 30 17:07:36.771562 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:36.771540 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-66f5sww7m"] Oct 30 17:07:37.809711 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:37.809679 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aeb6353a-f1de-4669-afd7-a17f36d2a194" path="/var/lib/kubelet/pods/aeb6353a-f1de-4669-afd7-a17f36d2a194/volumes" Oct 30 17:07:50.278221 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:50.278173 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 17:07:50.297648 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:07:50.297609 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:07:50.307051 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:50.307032 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 17:07:52.087025 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.086984 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm"] Oct 30 17:07:52.087811 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.087796 4861 cpu_manager.go:414] "containerMap: removing container" podUID="aeb6353a-f1de-4669-afd7-a17f36d2a194" containerName="payload-processing" Oct 30 17:07:52.087869 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.087812 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aeb6353a-f1de-4669-afd7-a17f36d2a194" containerName="payload-processing" Oct 30 17:07:52.087967 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.087939 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="aeb6353a-f1de-4669-afd7-a17f36d2a194" containerName="payload-processing" Oct 30 17:07:52.093112 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.093091 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:07:52.128328 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.128305 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"openshift-service-ca.crt\"" Oct 30 17:07:52.128489 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.128356 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d5b798\"" Oct 30 17:07:52.128489 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.128392 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"agc-cred-eda26afab71c\"" Oct 30 17:07:52.128489 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.128478 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"agc-cred-8dda90d8fcab\"" Oct 30 17:07:52.128739 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.128558 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"kube-root-ca.crt\"" Oct 30 17:07:52.128739 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.128641 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"payload-processing-external-model-e2e--02a9e3c4-dockercfg-cqwmp\"" Oct 30 17:07:52.128739 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.128701 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d5b798\"/\"routing-overlay\"" Oct 30 17:07:52.133163 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.133139 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm"] Oct 30 17:07:52.191244 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.191213 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl67c\" (UniqueName: \"kubernetes.io/projected/746b39b4-7b4b-4fe3-b9eb-944af7954669-kube-api-access-dl67c\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm\" (UID: \"746b39b4-7b4b-4fe3-b9eb-944af7954669\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:07:52.191416 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.191381 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/746b39b4-7b4b-4fe3-b9eb-944af7954669-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm\" (UID: \"746b39b4-7b4b-4fe3-b9eb-944af7954669\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:07:52.191520 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.191496 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/746b39b4-7b4b-4fe3-b9eb-944af7954669-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm\" (UID: \"746b39b4-7b4b-4fe3-b9eb-944af7954669\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:07:52.191625 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.191611 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/746b39b4-7b4b-4fe3-b9eb-944af7954669-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm\" (UID: \"746b39b4-7b4b-4fe3-b9eb-944af7954669\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:07:52.293052 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.293017 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/746b39b4-7b4b-4fe3-b9eb-944af7954669-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm\" (UID: \"746b39b4-7b4b-4fe3-b9eb-944af7954669\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:07:52.293235 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.293101 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/746b39b4-7b4b-4fe3-b9eb-944af7954669-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm\" (UID: \"746b39b4-7b4b-4fe3-b9eb-944af7954669\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:07:52.293235 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.293167 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/746b39b4-7b4b-4fe3-b9eb-944af7954669-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm\" (UID: \"746b39b4-7b4b-4fe3-b9eb-944af7954669\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:07:52.293235 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.293199 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl67c\" (UniqueName: \"kubernetes.io/projected/746b39b4-7b4b-4fe3-b9eb-944af7954669-kube-api-access-dl67c\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm\" (UID: \"746b39b4-7b4b-4fe3-b9eb-944af7954669\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:07:52.293660 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.293635 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/746b39b4-7b4b-4fe3-b9eb-944af7954669-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm\" (UID: \"746b39b4-7b4b-4fe3-b9eb-944af7954669\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:07:52.293830 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.293809 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/746b39b4-7b4b-4fe3-b9eb-944af7954669-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm\" (UID: \"746b39b4-7b4b-4fe3-b9eb-944af7954669\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:07:52.295895 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.295873 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/746b39b4-7b4b-4fe3-b9eb-944af7954669-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm\" (UID: \"746b39b4-7b4b-4fe3-b9eb-944af7954669\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:07:52.318297 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.318271 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl67c\" (UniqueName: \"kubernetes.io/projected/746b39b4-7b4b-4fe3-b9eb-944af7954669-kube-api-access-dl67c\") pod \"payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm\" (UID: \"746b39b4-7b4b-4fe3-b9eb-944af7954669\") " pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:07:52.408676 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.408603 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:07:52.726149 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.726121 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm"] Oct 30 17:07:52.727065 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:07:52.727035 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746b39b4_7b4b_4fe3_b9eb_944af7954669.slice/crio-e4bd0ad133a8cfc65814e3fcb2c296753d6ffb0d772a3b22037b952f18905207 WatchSource:0}: Error finding container e4bd0ad133a8cfc65814e3fcb2c296753d6ffb0d772a3b22037b952f18905207: Status 404 returned error can't find the container with id e4bd0ad133a8cfc65814e3fcb2c296753d6ffb0d772a3b22037b952f18905207 Oct 30 17:07:52.842691 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.842657 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" event={"ID":"746b39b4-7b4b-4fe3-b9eb-944af7954669","Type":"ContainerStarted","Data":"e98144ed1e26608d6875bdf32de224578d32ee6c4c1b12a1d19ffaf9cb59e420"} Oct 30 17:07:52.842903 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.842885 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:07:52.842994 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.842926 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" event={"ID":"746b39b4-7b4b-4fe3-b9eb-944af7954669","Type":"ContainerStarted","Data":"e4bd0ad133a8cfc65814e3fcb2c296753d6ffb0d772a3b22037b952f18905207"} Oct 30 17:07:52.891292 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:07:52.891245 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" podStartSLOduration=0.89123212 podStartE2EDuration="891.23212ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:52.887451511 +0000 UTC m=+2703.230125549" watchObservedRunningTime="2026-10-30 17:07:52.89123212 +0000 UTC m=+2703.233906158" Oct 30 17:08:03.851626 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:03.851597 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:08:20.290987 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:08:20.290950 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:08:30.741082 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:30.741002 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj"] Oct 30 17:08:35.741586 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:35.741543 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" podUID="f8bee04f-708a-4d98-bf9e-4d9d438b336b" containerName="payload-processing" containerID="cri-o://2e402a11ee3aeb4f2793720ef0a2feeca91b9c129e839421cf4e956d47fa20a5" gracePeriod=25 Oct 30 17:08:36.031213 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.031188 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:08:36.090090 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.090058 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8bee04f-708a-4d98-bf9e-4d9d438b336b-kube-api-access-cr6kw\" (UniqueName: \"kubernetes.io/projected/f8bee04f-708a-4d98-bf9e-4d9d438b336b-kube-api-access-cr6kw\") pod \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\" (UID: \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\") " Oct 30 17:08:36.090247 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.090204 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8bee04f-708a-4d98-bf9e-4d9d438b336b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8bee04f-708a-4d98-bf9e-4d9d438b336b-routing-overlay-volume\") pod \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\" (UID: \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\") " Oct 30 17:08:36.090615 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.090587 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bee04f-708a-4d98-bf9e-4d9d438b336b-routing-overlay-volume" pod "f8bee04f-708a-4d98-bf9e-4d9d438b336b" (UID: "f8bee04f-708a-4d98-bf9e-4d9d438b336b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:36.090772 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.090683 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8bee04f-708a-4d98-bf9e-4d9d438b336b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8bee04f-708a-4d98-bf9e-4d9d438b336b-provider-credentials\") pod \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\" (UID: \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\") " Oct 30 17:08:36.093338 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.092229 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8bee04f-708a-4d98-bf9e-4d9d438b336b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bee04f-708a-4d98-bf9e-4d9d438b336b-plugins-config-volume\") pod \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\" (UID: \"f8bee04f-708a-4d98-bf9e-4d9d438b336b\") " Oct 30 17:08:36.093338 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.092962 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8bee04f-708a-4d98-bf9e-4d9d438b336b-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:08:36.093338 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.093299 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bee04f-708a-4d98-bf9e-4d9d438b336b-plugins-config-volume" pod "f8bee04f-708a-4d98-bf9e-4d9d438b336b" (UID: "f8bee04f-708a-4d98-bf9e-4d9d438b336b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:36.093575 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.093362 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8bee04f-708a-4d98-bf9e-4d9d438b336b-kube-api-access-cr6kw" pod "f8bee04f-708a-4d98-bf9e-4d9d438b336b" (UID: "f8bee04f-708a-4d98-bf9e-4d9d438b336b"). InnerVolumeSpecName "kube-api-access-cr6kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:36.096829 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.096793 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8bee04f-708a-4d98-bf9e-4d9d438b336b-provider-credentials" pod "f8bee04f-708a-4d98-bf9e-4d9d438b336b" (UID: "f8bee04f-708a-4d98-bf9e-4d9d438b336b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:36.194361 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.194320 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr6kw\" (UniqueName: \"kubernetes.io/projected/f8bee04f-708a-4d98-bf9e-4d9d438b336b-kube-api-access-cr6kw\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:08:36.194361 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.194353 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8bee04f-708a-4d98-bf9e-4d9d438b336b-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:08:36.194361 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.194366 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8bee04f-708a-4d98-bf9e-4d9d438b336b-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:08:36.235551 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.235514 4861 generic.go:356] "Generic (PLEG): container finished" podID="f8bee04f-708a-4d98-bf9e-4d9d438b336b" containerID="2e402a11ee3aeb4f2793720ef0a2feeca91b9c129e839421cf4e956d47fa20a5" exitCode=0 Oct 30 17:08:36.235885 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.235570 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" Oct 30 17:08:36.236024 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.235572 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" event={"ID":"f8bee04f-708a-4d98-bf9e-4d9d438b336b","Type":"ContainerDied","Data":"2e402a11ee3aeb4f2793720ef0a2feeca91b9c129e839421cf4e956d47fa20a5"} Oct 30 17:08:36.236100 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.236060 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj" event={"ID":"f8bee04f-708a-4d98-bf9e-4d9d438b336b","Type":"ContainerDied","Data":"e2c946a8703d54a1a24794694bc596d0b91faf0daa14126e8a7fa7cd423bf47b"} Oct 30 17:08:36.236157 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.236097 4861 scope.go:122] "RemoveContainer" containerID="2e402a11ee3aeb4f2793720ef0a2feeca91b9c129e839421cf4e956d47fa20a5" Oct 30 17:08:36.253151 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.253126 4861 scope.go:122] "RemoveContainer" containerID="2e402a11ee3aeb4f2793720ef0a2feeca91b9c129e839421cf4e956d47fa20a5" Oct 30 17:08:36.253556 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:08:36.253507 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e402a11ee3aeb4f2793720ef0a2feeca91b9c129e839421cf4e956d47fa20a5\": container with ID starting with 2e402a11ee3aeb4f2793720ef0a2feeca91b9c129e839421cf4e956d47fa20a5 not found: ID does not exist" containerID="2e402a11ee3aeb4f2793720ef0a2feeca91b9c129e839421cf4e956d47fa20a5" Oct 30 17:08:36.253677 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.253549 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e402a11ee3aeb4f2793720ef0a2feeca91b9c129e839421cf4e956d47fa20a5"} err="failed to get container status \"2e402a11ee3aeb4f2793720ef0a2feeca91b9c129e839421cf4e956d47fa20a5\": rpc error: code = NotFound desc = could not find container \"2e402a11ee3aeb4f2793720ef0a2feeca91b9c129e839421cf4e956d47fa20a5\": container with ID starting with 2e402a11ee3aeb4f2793720ef0a2feeca91b9c129e839421cf4e956d47fa20a5 not found: ID does not exist" Oct 30 17:08:36.301924 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.301894 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj"] Oct 30 17:08:36.305503 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:36.305467 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-8685jqlfj"] Oct 30 17:08:37.809184 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:37.809143 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8bee04f-708a-4d98-bf9e-4d9d438b336b" path="/var/lib/kubelet/pods/f8bee04f-708a-4d98-bf9e-4d9d438b336b/volumes" Oct 30 17:08:41.476948 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.476918 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb"] Oct 30 17:08:41.477709 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.477694 4861 cpu_manager.go:414] "containerMap: removing container" podUID="f8bee04f-708a-4d98-bf9e-4d9d438b336b" containerName="payload-processing" Oct 30 17:08:41.477823 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.477710 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8bee04f-708a-4d98-bf9e-4d9d438b336b" containerName="payload-processing" Oct 30 17:08:41.477898 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.477888 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8bee04f-708a-4d98-bf9e-4d9d438b336b" containerName="payload-processing" Oct 30 17:08:41.482849 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.482831 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:08:41.502071 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.502044 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb"] Oct 30 17:08:41.510987 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.510964 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"agc-cred-cf0331906c37\"" Oct 30 17:08:41.511169 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.510964 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"agc-cred-af511487c8db\"" Oct 30 17:08:41.511312 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.511173 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"kube-root-ca.crt\"" Oct 30 17:08:41.511312 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.511228 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"openshift-service-ca.crt\"" Oct 30 17:08:41.511623 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.511581 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"payload-processing-external-model-e2e--e2ae9620-dockercfg-rjgsg\"" Oct 30 17:08:41.513443 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.513423 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"routing-overlay\"" Oct 30 17:08:41.513643 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.513427 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-43197b\"" Oct 30 17:08:41.589522 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.589489 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lpzx\" (UniqueName: \"kubernetes.io/projected/ebc0fdde-2b32-4418-a05b-da30146a85a0-kube-api-access-8lpzx\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-566877xxb\" (UID: \"ebc0fdde-2b32-4418-a05b-da30146a85a0\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:08:41.589674 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.589544 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ebc0fdde-2b32-4418-a05b-da30146a85a0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-566877xxb\" (UID: \"ebc0fdde-2b32-4418-a05b-da30146a85a0\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:08:41.589816 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.589798 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ebc0fdde-2b32-4418-a05b-da30146a85a0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-566877xxb\" (UID: \"ebc0fdde-2b32-4418-a05b-da30146a85a0\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:08:41.589863 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.589826 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc0fdde-2b32-4418-a05b-da30146a85a0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-566877xxb\" (UID: \"ebc0fdde-2b32-4418-a05b-da30146a85a0\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:08:41.691367 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.691336 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lpzx\" (UniqueName: \"kubernetes.io/projected/ebc0fdde-2b32-4418-a05b-da30146a85a0-kube-api-access-8lpzx\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-566877xxb\" (UID: \"ebc0fdde-2b32-4418-a05b-da30146a85a0\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:08:41.691539 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.691391 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ebc0fdde-2b32-4418-a05b-da30146a85a0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-566877xxb\" (UID: \"ebc0fdde-2b32-4418-a05b-da30146a85a0\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:08:41.691539 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.691533 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ebc0fdde-2b32-4418-a05b-da30146a85a0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-566877xxb\" (UID: \"ebc0fdde-2b32-4418-a05b-da30146a85a0\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:08:41.691625 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.691552 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc0fdde-2b32-4418-a05b-da30146a85a0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-566877xxb\" (UID: \"ebc0fdde-2b32-4418-a05b-da30146a85a0\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:08:41.692252 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.692230 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ebc0fdde-2b32-4418-a05b-da30146a85a0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-566877xxb\" (UID: \"ebc0fdde-2b32-4418-a05b-da30146a85a0\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:08:41.692329 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.692317 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc0fdde-2b32-4418-a05b-da30146a85a0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-566877xxb\" (UID: \"ebc0fdde-2b32-4418-a05b-da30146a85a0\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:08:41.694236 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.694216 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ebc0fdde-2b32-4418-a05b-da30146a85a0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-566877xxb\" (UID: \"ebc0fdde-2b32-4418-a05b-da30146a85a0\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:08:41.716657 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.716635 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lpzx\" (UniqueName: \"kubernetes.io/projected/ebc0fdde-2b32-4418-a05b-da30146a85a0-kube-api-access-8lpzx\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-566877xxb\" (UID: \"ebc0fdde-2b32-4418-a05b-da30146a85a0\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:08:41.798254 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:41.798154 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:08:42.113103 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:42.113071 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb"] Oct 30 17:08:42.282483 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:42.282451 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" event={"ID":"ebc0fdde-2b32-4418-a05b-da30146a85a0","Type":"ContainerStarted","Data":"e322154633887ee4da17616db2d3bf8ee964bc65303d63696b4f36f915f1f156"} Oct 30 17:08:42.282698 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:42.282504 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" event={"ID":"ebc0fdde-2b32-4418-a05b-da30146a85a0","Type":"ContainerStarted","Data":"7163ad0f64428dd7ee0d94cdecfe5f02fe057302917143729490749e40d5cf5f"} Oct 30 17:08:42.282698 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:42.282552 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:08:42.328878 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:42.328818 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" podStartSLOduration=1.328804732 podStartE2EDuration="1.328804732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:42.328211022 +0000 UTC m=+2752.670885061" watchObservedRunningTime="2026-10-30 17:08:42.328804732 +0000 UTC m=+2752.671478769" Oct 30 17:08:50.293703 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:08:50.293664 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:08:53.181138 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:53.181105 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm"] Oct 30 17:08:53.297476 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:53.297446 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:08:58.182160 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:58.182119 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" podUID="3468a16e-dad6-45b2-b516-dc8bf6c86ebd" containerName="payload-processing" containerID="cri-o://9e170f553ae71d4daab3661f9a49d302f8f23d4c8a59a831d640944fa1aeffd9" gracePeriod=25 Oct 30 17:08:58.412181 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:58.412151 4861 generic.go:356] "Generic (PLEG): container finished" podID="3468a16e-dad6-45b2-b516-dc8bf6c86ebd" containerID="9e170f553ae71d4daab3661f9a49d302f8f23d4c8a59a831d640944fa1aeffd9" exitCode=0 Oct 30 17:08:58.412345 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:58.412334 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" event={"ID":"3468a16e-dad6-45b2-b516-dc8bf6c86ebd","Type":"ContainerDied","Data":"9e170f553ae71d4daab3661f9a49d302f8f23d4c8a59a831d640944fa1aeffd9"} Oct 30 17:08:58.491186 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:58.491160 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:08:58.655159 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:58.655128 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-plugins-config-volume\") pod \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\" (UID: \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\") " Oct 30 17:08:58.655336 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:58.655188 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-kube-api-access-49ptg\" (UniqueName: \"kubernetes.io/projected/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-kube-api-access-49ptg\") pod \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\" (UID: \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\") " Oct 30 17:08:58.655336 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:58.655228 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-routing-overlay-volume\") pod \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\" (UID: \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\") " Oct 30 17:08:58.655336 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:58.655257 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-provider-credentials\") pod \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\" (UID: \"3468a16e-dad6-45b2-b516-dc8bf6c86ebd\") " Oct 30 17:08:58.655591 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:58.655559 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-plugins-config-volume" pod "3468a16e-dad6-45b2-b516-dc8bf6c86ebd" (UID: "3468a16e-dad6-45b2-b516-dc8bf6c86ebd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:58.655758 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:58.655671 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-routing-overlay-volume" pod "3468a16e-dad6-45b2-b516-dc8bf6c86ebd" (UID: "3468a16e-dad6-45b2-b516-dc8bf6c86ebd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:58.656035 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:58.655977 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:08:58.656035 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:58.656006 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:08:58.657840 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:58.657810 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-provider-credentials" pod "3468a16e-dad6-45b2-b516-dc8bf6c86ebd" (UID: "3468a16e-dad6-45b2-b516-dc8bf6c86ebd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:58.657926 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:58.657901 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-kube-api-access-49ptg" pod "3468a16e-dad6-45b2-b516-dc8bf6c86ebd" (UID: "3468a16e-dad6-45b2-b516-dc8bf6c86ebd"). InnerVolumeSpecName "kube-api-access-49ptg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:58.756771 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:58.756746 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49ptg\" (UniqueName: \"kubernetes.io/projected/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-kube-api-access-49ptg\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:08:58.756771 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:58.756770 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3468a16e-dad6-45b2-b516-dc8bf6c86ebd-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:08:59.421029 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:59.420997 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" Oct 30 17:08:59.421438 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:59.421035 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm" event={"ID":"3468a16e-dad6-45b2-b516-dc8bf6c86ebd","Type":"ContainerDied","Data":"a1195756f82f47d4eb244788d5b985d40b8cbd9356b05f07cb23df7870f76ee9"} Oct 30 17:08:59.421438 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:59.421099 4861 scope.go:122] "RemoveContainer" containerID="9e170f553ae71d4daab3661f9a49d302f8f23d4c8a59a831d640944fa1aeffd9" Oct 30 17:08:59.480033 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:59.480005 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm"] Oct 30 17:08:59.493864 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:59.493842 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7fd7xs4vm"] Oct 30 17:08:59.810814 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:08:59.810781 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3468a16e-dad6-45b2-b516-dc8bf6c86ebd" path="/var/lib/kubelet/pods/3468a16e-dad6-45b2-b516-dc8bf6c86ebd/volumes" Oct 30 17:09:03.723922 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.723877 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2"] Oct 30 17:09:03.724817 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.724796 4861 cpu_manager.go:414] "containerMap: removing container" podUID="3468a16e-dad6-45b2-b516-dc8bf6c86ebd" containerName="payload-processing" Oct 30 17:09:03.724817 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.724816 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3468a16e-dad6-45b2-b516-dc8bf6c86ebd" containerName="payload-processing" Oct 30 17:09:03.724986 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.724963 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="3468a16e-dad6-45b2-b516-dc8bf6c86ebd" containerName="payload-processing" Oct 30 17:09:03.730012 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.729993 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:03.750334 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.750298 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2"] Oct 30 17:09:03.824386 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.824351 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67928c83-07ad-4882-8135-ed566ad911f3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2\" (UID: \"67928c83-07ad-4882-8135-ed566ad911f3\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:03.824570 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.824519 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67928c83-07ad-4882-8135-ed566ad911f3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2\" (UID: \"67928c83-07ad-4882-8135-ed566ad911f3\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:03.824657 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.824639 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrkvg\" (UniqueName: \"kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-kube-api-access-hrkvg\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2\" (UID: \"67928c83-07ad-4882-8135-ed566ad911f3\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:03.824711 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.824675 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2\" (UID: \"67928c83-07ad-4882-8135-ed566ad911f3\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:03.925857 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.925821 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkvg\" (UniqueName: \"kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-kube-api-access-hrkvg\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2\" (UID: \"67928c83-07ad-4882-8135-ed566ad911f3\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:03.925857 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.925860 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2\" (UID: \"67928c83-07ad-4882-8135-ed566ad911f3\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:03.926108 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.926004 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67928c83-07ad-4882-8135-ed566ad911f3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2\" (UID: \"67928c83-07ad-4882-8135-ed566ad911f3\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:03.926108 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:03.926046 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3106d6/agc-cred-129cce3672c2: secret "agc-cred-129cce3672c2" not found Oct 30 17:09:03.926108 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:03.926071 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2: secret "agc-cred-129cce3672c2" not found Oct 30 17:09:03.926236 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:03.926135 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-provider-credentials podName:67928c83-07ad-4882-8135-ed566ad911f3 nodeName:}" failed. No retries permitted until 2026-10-30 17:09:04.426116791 +0000 UTC m=+2774.768790831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" (UID: "67928c83-07ad-4882-8135-ed566ad911f3") : secret "agc-cred-129cce3672c2" not found Oct 30 17:09:03.926236 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.926161 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67928c83-07ad-4882-8135-ed566ad911f3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2\" (UID: \"67928c83-07ad-4882-8135-ed566ad911f3\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:03.926689 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.926669 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67928c83-07ad-4882-8135-ed566ad911f3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2\" (UID: \"67928c83-07ad-4882-8135-ed566ad911f3\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:03.926894 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.926875 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67928c83-07ad-4882-8135-ed566ad911f3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2\" (UID: \"67928c83-07ad-4882-8135-ed566ad911f3\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:03.954369 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:03.954343 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrkvg\" (UniqueName: \"kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-kube-api-access-hrkvg\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2\" (UID: \"67928c83-07ad-4882-8135-ed566ad911f3\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:04.433662 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:04.433628 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2\" (UID: \"67928c83-07ad-4882-8135-ed566ad911f3\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:04.433857 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:04.433801 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3106d6/agc-cred-129cce3672c2: secret "agc-cred-129cce3672c2" not found Oct 30 17:09:04.433857 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:04.433826 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2: secret "agc-cred-129cce3672c2" not found Oct 30 17:09:04.433943 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:04.433889 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-provider-credentials podName:67928c83-07ad-4882-8135-ed566ad911f3 nodeName:}" failed. No retries permitted until 2026-10-30 17:09:05.43387546 +0000 UTC m=+2775.776549479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" (UID: "67928c83-07ad-4882-8135-ed566ad911f3") : secret "agc-cred-129cce3672c2" not found Oct 30 17:09:05.299015 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:05.298968 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2"] Oct 30 17:09:05.299504 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:05.299442 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" podUID="67928c83-07ad-4882-8135-ed566ad911f3" Oct 30 17:09:05.446113 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:05.446077 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2\" (UID: \"67928c83-07ad-4882-8135-ed566ad911f3\") " pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:05.446260 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:05.446218 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3106d6/agc-cred-129cce3672c2: secret "agc-cred-129cce3672c2" not found Oct 30 17:09:05.446260 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:05.446241 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2: secret "agc-cred-129cce3672c2" not found Oct 30 17:09:05.446358 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:05.446316 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-provider-credentials podName:67928c83-07ad-4882-8135-ed566ad911f3 nodeName:}" failed. No retries permitted until 2026-10-30 17:09:07.446300757 +0000 UTC m=+2777.788974798 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" (UID: "67928c83-07ad-4882-8135-ed566ad911f3") : secret "agc-cred-129cce3672c2" not found Oct 30 17:09:05.468925 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:05.468898 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:05.477938 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:05.477917 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:05.547744 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:05.547697 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67928c83-07ad-4882-8135-ed566ad911f3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67928c83-07ad-4882-8135-ed566ad911f3-routing-overlay-volume\") pod \"67928c83-07ad-4882-8135-ed566ad911f3\" (UID: \"67928c83-07ad-4882-8135-ed566ad911f3\") " Oct 30 17:09:05.547877 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:05.547764 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-kube-api-access-hrkvg\" (UniqueName: \"kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-kube-api-access-hrkvg\") pod \"67928c83-07ad-4882-8135-ed566ad911f3\" (UID: \"67928c83-07ad-4882-8135-ed566ad911f3\") " Oct 30 17:09:05.547877 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:05.547835 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67928c83-07ad-4882-8135-ed566ad911f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67928c83-07ad-4882-8135-ed566ad911f3-plugins-config-volume\") pod \"67928c83-07ad-4882-8135-ed566ad911f3\" (UID: \"67928c83-07ad-4882-8135-ed566ad911f3\") " Oct 30 17:09:05.548097 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:05.548073 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67928c83-07ad-4882-8135-ed566ad911f3-routing-overlay-volume" pod "67928c83-07ad-4882-8135-ed566ad911f3" (UID: "67928c83-07ad-4882-8135-ed566ad911f3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:05.548252 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:05.548231 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67928c83-07ad-4882-8135-ed566ad911f3-plugins-config-volume" pod "67928c83-07ad-4882-8135-ed566ad911f3" (UID: "67928c83-07ad-4882-8135-ed566ad911f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:05.548337 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:05.548305 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67928c83-07ad-4882-8135-ed566ad911f3-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:09:05.550140 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:05.550100 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-kube-api-access-hrkvg" pod "67928c83-07ad-4882-8135-ed566ad911f3" (UID: "67928c83-07ad-4882-8135-ed566ad911f3"). InnerVolumeSpecName "kube-api-access-hrkvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:05.649386 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:05.649363 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrkvg\" (UniqueName: \"kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-kube-api-access-hrkvg\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:09:05.649386 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:05.649387 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67928c83-07ad-4882-8135-ed566ad911f3-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:09:06.476228 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:06.476190 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2" Oct 30 17:09:06.565181 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:06.565151 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2"] Oct 30 17:09:06.569185 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:06.569154 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-785cj6vl2"] Oct 30 17:09:06.663342 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:06.663316 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67928c83-07ad-4882-8135-ed566ad911f3-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:09:07.811085 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:07.811044 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67928c83-07ad-4882-8135-ed566ad911f3" path="/var/lib/kubelet/pods/67928c83-07ad-4882-8135-ed566ad911f3/volumes" Oct 30 17:09:16.908775 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:16.908716 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8"] Oct 30 17:09:20.292061 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:20.292018 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:09:21.909924 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:21.909883 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" podUID="a34df277-cd10-4cc9-ad56-5ebf552938f5" containerName="payload-processing" containerID="cri-o://a6e38680a0757acb253609e3274556992191cbe59cf1276c63e77d9a3abe6ab4" gracePeriod=25 Oct 30 17:09:22.196551 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.196519 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:09:22.360423 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.360391 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a34df277-cd10-4cc9-ad56-5ebf552938f5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a34df277-cd10-4cc9-ad56-5ebf552938f5-provider-credentials\") pod \"a34df277-cd10-4cc9-ad56-5ebf552938f5\" (UID: \"a34df277-cd10-4cc9-ad56-5ebf552938f5\") " Oct 30 17:09:22.360607 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.360457 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a34df277-cd10-4cc9-ad56-5ebf552938f5-kube-api-access-zc5rq\" (UniqueName: \"kubernetes.io/projected/a34df277-cd10-4cc9-ad56-5ebf552938f5-kube-api-access-zc5rq\") pod \"a34df277-cd10-4cc9-ad56-5ebf552938f5\" (UID: \"a34df277-cd10-4cc9-ad56-5ebf552938f5\") " Oct 30 17:09:22.360607 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.360512 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a34df277-cd10-4cc9-ad56-5ebf552938f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34df277-cd10-4cc9-ad56-5ebf552938f5-plugins-config-volume\") pod \"a34df277-cd10-4cc9-ad56-5ebf552938f5\" (UID: \"a34df277-cd10-4cc9-ad56-5ebf552938f5\") " Oct 30 17:09:22.360702 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.360614 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a34df277-cd10-4cc9-ad56-5ebf552938f5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a34df277-cd10-4cc9-ad56-5ebf552938f5-routing-overlay-volume\") pod \"a34df277-cd10-4cc9-ad56-5ebf552938f5\" (UID: \"a34df277-cd10-4cc9-ad56-5ebf552938f5\") " Oct 30 17:09:22.360998 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.360968 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34df277-cd10-4cc9-ad56-5ebf552938f5-plugins-config-volume" pod "a34df277-cd10-4cc9-ad56-5ebf552938f5" (UID: "a34df277-cd10-4cc9-ad56-5ebf552938f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:22.360998 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.360993 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34df277-cd10-4cc9-ad56-5ebf552938f5-routing-overlay-volume" pod "a34df277-cd10-4cc9-ad56-5ebf552938f5" (UID: "a34df277-cd10-4cc9-ad56-5ebf552938f5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:22.361378 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.361309 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a34df277-cd10-4cc9-ad56-5ebf552938f5-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:09:22.361378 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.361332 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a34df277-cd10-4cc9-ad56-5ebf552938f5-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:09:22.362974 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.362947 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34df277-cd10-4cc9-ad56-5ebf552938f5-kube-api-access-zc5rq" pod "a34df277-cd10-4cc9-ad56-5ebf552938f5" (UID: "a34df277-cd10-4cc9-ad56-5ebf552938f5"). InnerVolumeSpecName "kube-api-access-zc5rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:22.363095 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.363019 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34df277-cd10-4cc9-ad56-5ebf552938f5-provider-credentials" pod "a34df277-cd10-4cc9-ad56-5ebf552938f5" (UID: "a34df277-cd10-4cc9-ad56-5ebf552938f5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:22.462269 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.462196 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a34df277-cd10-4cc9-ad56-5ebf552938f5-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:09:22.462269 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.462221 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc5rq\" (UniqueName: \"kubernetes.io/projected/a34df277-cd10-4cc9-ad56-5ebf552938f5-kube-api-access-zc5rq\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:09:22.610389 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.610355 4861 generic.go:356] "Generic (PLEG): container finished" podID="a34df277-cd10-4cc9-ad56-5ebf552938f5" containerID="a6e38680a0757acb253609e3274556992191cbe59cf1276c63e77d9a3abe6ab4" exitCode=0 Oct 30 17:09:22.610565 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.610406 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" Oct 30 17:09:22.610565 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.610427 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" event={"ID":"a34df277-cd10-4cc9-ad56-5ebf552938f5","Type":"ContainerDied","Data":"a6e38680a0757acb253609e3274556992191cbe59cf1276c63e77d9a3abe6ab4"} Oct 30 17:09:22.610565 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.610480 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8" event={"ID":"a34df277-cd10-4cc9-ad56-5ebf552938f5","Type":"ContainerDied","Data":"11c0ca2441a15503dd35ec47e06b2f5da6ea77215751db314e3f145061e31547"} Oct 30 17:09:22.610565 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.610507 4861 scope.go:122] "RemoveContainer" containerID="a6e38680a0757acb253609e3274556992191cbe59cf1276c63e77d9a3abe6ab4" Oct 30 17:09:22.622157 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.622140 4861 scope.go:122] "RemoveContainer" containerID="a6e38680a0757acb253609e3274556992191cbe59cf1276c63e77d9a3abe6ab4" Oct 30 17:09:22.622439 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:22.622419 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e38680a0757acb253609e3274556992191cbe59cf1276c63e77d9a3abe6ab4\": container with ID starting with a6e38680a0757acb253609e3274556992191cbe59cf1276c63e77d9a3abe6ab4 not found: ID does not exist" containerID="a6e38680a0757acb253609e3274556992191cbe59cf1276c63e77d9a3abe6ab4" Oct 30 17:09:22.622491 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.622448 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e38680a0757acb253609e3274556992191cbe59cf1276c63e77d9a3abe6ab4"} err="failed to get container status \"a6e38680a0757acb253609e3274556992191cbe59cf1276c63e77d9a3abe6ab4\": rpc error: code = NotFound desc = could not find container \"a6e38680a0757acb253609e3274556992191cbe59cf1276c63e77d9a3abe6ab4\": container with ID starting with a6e38680a0757acb253609e3274556992191cbe59cf1276c63e77d9a3abe6ab4 not found: ID does not exist" Oct 30 17:09:22.670637 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.670613 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8"] Oct 30 17:09:22.674102 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:22.674079 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3106d6/payload-processing-external-model-e2e-extmod-a-3106d6-7dddkwwj8"] Oct 30 17:09:23.809972 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:23.809942 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a34df277-cd10-4cc9-ad56-5ebf552938f5" path="/var/lib/kubelet/pods/a34df277-cd10-4cc9-ad56-5ebf552938f5/volumes" Oct 30 17:09:31.939756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:31.939706 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w"] Oct 30 17:09:31.942421 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:31.940485 4861 cpu_manager.go:414] "containerMap: removing container" podUID="a34df277-cd10-4cc9-ad56-5ebf552938f5" containerName="payload-processing" Oct 30 17:09:31.942421 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:31.940497 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a34df277-cd10-4cc9-ad56-5ebf552938f5" containerName="payload-processing" Oct 30 17:09:31.942421 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:31.940610 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="a34df277-cd10-4cc9-ad56-5ebf552938f5" containerName="payload-processing" Oct 30 17:09:31.944389 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:31.944367 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:09:31.965455 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:31.965433 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w"] Oct 30 17:09:32.013350 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.013317 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mqhf\" (UniqueName: \"kubernetes.io/projected/9971fa48-be67-4483-9584-3e96b8bfb84c-kube-api-access-2mqhf\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w\" (UID: \"9971fa48-be67-4483-9584-3e96b8bfb84c\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:09:32.013508 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.013361 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9971fa48-be67-4483-9584-3e96b8bfb84c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w\" (UID: \"9971fa48-be67-4483-9584-3e96b8bfb84c\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:09:32.013583 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.013562 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9971fa48-be67-4483-9584-3e96b8bfb84c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w\" (UID: \"9971fa48-be67-4483-9584-3e96b8bfb84c\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:09:32.013623 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.013595 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9971fa48-be67-4483-9584-3e96b8bfb84c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w\" (UID: \"9971fa48-be67-4483-9584-3e96b8bfb84c\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:09:32.115453 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.115422 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mqhf\" (UniqueName: \"kubernetes.io/projected/9971fa48-be67-4483-9584-3e96b8bfb84c-kube-api-access-2mqhf\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w\" (UID: \"9971fa48-be67-4483-9584-3e96b8bfb84c\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:09:32.115619 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.115466 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9971fa48-be67-4483-9584-3e96b8bfb84c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w\" (UID: \"9971fa48-be67-4483-9584-3e96b8bfb84c\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:09:32.115619 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.115592 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9971fa48-be67-4483-9584-3e96b8bfb84c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w\" (UID: \"9971fa48-be67-4483-9584-3e96b8bfb84c\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:09:32.115800 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.115683 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9971fa48-be67-4483-9584-3e96b8bfb84c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w\" (UID: \"9971fa48-be67-4483-9584-3e96b8bfb84c\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:09:32.116509 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.116481 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9971fa48-be67-4483-9584-3e96b8bfb84c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w\" (UID: \"9971fa48-be67-4483-9584-3e96b8bfb84c\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:09:32.116644 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.116520 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9971fa48-be67-4483-9584-3e96b8bfb84c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w\" (UID: \"9971fa48-be67-4483-9584-3e96b8bfb84c\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:09:32.118686 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.118655 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9971fa48-be67-4483-9584-3e96b8bfb84c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w\" (UID: \"9971fa48-be67-4483-9584-3e96b8bfb84c\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:09:32.140512 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.140492 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mqhf\" (UniqueName: \"kubernetes.io/projected/9971fa48-be67-4483-9584-3e96b8bfb84c-kube-api-access-2mqhf\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w\" (UID: \"9971fa48-be67-4483-9584-3e96b8bfb84c\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:09:32.259844 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.259805 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:09:32.613234 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.606194 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w"] Oct 30 17:09:32.633630 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:09:32.633590 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9971fa48_be67_4483_9584_3e96b8bfb84c.slice/crio-8d028c4a5d4250af41aa7f52be2602dfc97e3642ff1572941329dc5aa7f3eeef WatchSource:0}: Error finding container 8d028c4a5d4250af41aa7f52be2602dfc97e3642ff1572941329dc5aa7f3eeef: Status 404 returned error can't find the container with id 8d028c4a5d4250af41aa7f52be2602dfc97e3642ff1572941329dc5aa7f3eeef Oct 30 17:09:32.733182 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.733126 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" event={"ID":"9971fa48-be67-4483-9584-3e96b8bfb84c","Type":"ContainerStarted","Data":"3fafef9bb3a3aea542a3aaf0dd4cbddc1647c2780fe4fff255a9df60d45e2154"} Oct 30 17:09:32.733518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.733490 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:09:32.733633 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.733539 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" event={"ID":"9971fa48-be67-4483-9584-3e96b8bfb84c","Type":"ContainerStarted","Data":"8d028c4a5d4250af41aa7f52be2602dfc97e3642ff1572941329dc5aa7f3eeef"} Oct 30 17:09:32.784467 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:32.784422 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" podStartSLOduration=1.784407677 podStartE2EDuration="1.784407677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:32.781446348 +0000 UTC m=+2803.124120387" watchObservedRunningTime="2026-10-30 17:09:32.784407677 +0000 UTC m=+2803.127081717" Oct 30 17:09:33.993041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:33.993006 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r"] Oct 30 17:09:33.997918 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:33.997900 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:09:34.038111 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.038076 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f590f7630637\"" Oct 30 17:09:34.041111 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.040787 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 17:09:34.041111 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.040861 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pmvgh\"" Oct 30 17:09:34.041111 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.040968 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 17:09:34.041111 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.041066 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 17:09:34.043949 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.041129 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 17:09:34.043949 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.041990 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b17d7937-c70a-49ae-ae78-8b277d1d39fb-plugins-config-volume\") pod \"payload-processing-external-model-5b4f7df67-twr2r\" (UID: \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\") " pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:09:34.043949 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.042075 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xvd4\" (UniqueName: \"kubernetes.io/projected/b17d7937-c70a-49ae-ae78-8b277d1d39fb-kube-api-access-5xvd4\") pod \"payload-processing-external-model-5b4f7df67-twr2r\" (UID: \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\") " pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:09:34.043949 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.042122 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b17d7937-c70a-49ae-ae78-8b277d1d39fb-routing-overlay-volume\") pod \"payload-processing-external-model-5b4f7df67-twr2r\" (UID: \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\") " pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:09:34.043949 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.042250 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b17d7937-c70a-49ae-ae78-8b277d1d39fb-provider-credentials\") pod \"payload-processing-external-model-5b4f7df67-twr2r\" (UID: \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\") " pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:09:34.046218 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.046187 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r"] Oct 30 17:09:34.144420 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.144367 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b17d7937-c70a-49ae-ae78-8b277d1d39fb-plugins-config-volume\") pod \"payload-processing-external-model-5b4f7df67-twr2r\" (UID: \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\") " pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:09:34.144661 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.144476 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xvd4\" (UniqueName: \"kubernetes.io/projected/b17d7937-c70a-49ae-ae78-8b277d1d39fb-kube-api-access-5xvd4\") pod \"payload-processing-external-model-5b4f7df67-twr2r\" (UID: \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\") " pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:09:34.144661 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.144538 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b17d7937-c70a-49ae-ae78-8b277d1d39fb-routing-overlay-volume\") pod \"payload-processing-external-model-5b4f7df67-twr2r\" (UID: \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\") " pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:09:34.144805 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.144681 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b17d7937-c70a-49ae-ae78-8b277d1d39fb-provider-credentials\") pod \"payload-processing-external-model-5b4f7df67-twr2r\" (UID: \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\") " pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:09:34.145482 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.145449 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b17d7937-c70a-49ae-ae78-8b277d1d39fb-plugins-config-volume\") pod \"payload-processing-external-model-5b4f7df67-twr2r\" (UID: \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\") " pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:09:34.145789 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.145764 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b17d7937-c70a-49ae-ae78-8b277d1d39fb-routing-overlay-volume\") pod \"payload-processing-external-model-5b4f7df67-twr2r\" (UID: \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\") " pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:09:34.148152 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.148127 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b17d7937-c70a-49ae-ae78-8b277d1d39fb-provider-credentials\") pod \"payload-processing-external-model-5b4f7df67-twr2r\" (UID: \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\") " pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:09:34.170332 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.170299 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xvd4\" (UniqueName: \"kubernetes.io/projected/b17d7937-c70a-49ae-ae78-8b277d1d39fb-kube-api-access-5xvd4\") pod \"payload-processing-external-model-5b4f7df67-twr2r\" (UID: \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\") " pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:09:34.313440 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.313361 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:09:34.622281 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.622247 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r"] Oct 30 17:09:34.623141 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:09:34.623116 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17d7937_c70a_49ae_ae78_8b277d1d39fb.slice/crio-0d1a544bd2ff9bb38612c26cb8bc700b420c8b9a961b09814a897d263263491b WatchSource:0}: Error finding container 0d1a544bd2ff9bb38612c26cb8bc700b420c8b9a961b09814a897d263263491b: Status 404 returned error can't find the container with id 0d1a544bd2ff9bb38612c26cb8bc700b420c8b9a961b09814a897d263263491b Oct 30 17:09:34.749687 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.749650 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" event={"ID":"b17d7937-c70a-49ae-ae78-8b277d1d39fb","Type":"ContainerStarted","Data":"e6e682b1d701a0d51ddda0a359680784f9ea31c603bad6b38606fb7f7de2f879"} Oct 30 17:09:34.749879 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.749709 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:09:34.749879 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.749741 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" event={"ID":"b17d7937-c70a-49ae-ae78-8b277d1d39fb","Type":"ContainerStarted","Data":"0d1a544bd2ff9bb38612c26cb8bc700b420c8b9a961b09814a897d263263491b"} Oct 30 17:09:34.794419 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:34.794376 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" podStartSLOduration=1.79436412 podStartE2EDuration="1.79436412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:34.794144693 +0000 UTC m=+2805.136818731" watchObservedRunningTime="2026-10-30 17:09:34.79436412 +0000 UTC m=+2805.137038157" Oct 30 17:09:43.743924 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:43.743895 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:09:43.837837 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:43.837805 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb"] Oct 30 17:09:45.761279 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:45.761251 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:09:48.838334 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:48.838290 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" podUID="ebc0fdde-2b32-4418-a05b-da30146a85a0" containerName="payload-processing" containerID="cri-o://e322154633887ee4da17616db2d3bf8ee964bc65303d63696b4f36f915f1f156" gracePeriod=25 Oct 30 17:09:49.116663 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.116637 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:09:49.157781 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.157717 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc0fdde-2b32-4418-a05b-da30146a85a0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ebc0fdde-2b32-4418-a05b-da30146a85a0-provider-credentials\") pod \"ebc0fdde-2b32-4418-a05b-da30146a85a0\" (UID: \"ebc0fdde-2b32-4418-a05b-da30146a85a0\") " Oct 30 17:09:49.157916 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.157840 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebc0fdde-2b32-4418-a05b-da30146a85a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc0fdde-2b32-4418-a05b-da30146a85a0-plugins-config-volume\") pod \"ebc0fdde-2b32-4418-a05b-da30146a85a0\" (UID: \"ebc0fdde-2b32-4418-a05b-da30146a85a0\") " Oct 30 17:09:49.157916 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.157868 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebc0fdde-2b32-4418-a05b-da30146a85a0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ebc0fdde-2b32-4418-a05b-da30146a85a0-routing-overlay-volume\") pod \"ebc0fdde-2b32-4418-a05b-da30146a85a0\" (UID: \"ebc0fdde-2b32-4418-a05b-da30146a85a0\") " Oct 30 17:09:49.157916 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.157903 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc0fdde-2b32-4418-a05b-da30146a85a0-kube-api-access-8lpzx\" (UniqueName: \"kubernetes.io/projected/ebc0fdde-2b32-4418-a05b-da30146a85a0-kube-api-access-8lpzx\") pod \"ebc0fdde-2b32-4418-a05b-da30146a85a0\" (UID: \"ebc0fdde-2b32-4418-a05b-da30146a85a0\") " Oct 30 17:09:49.158307 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.158278 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc0fdde-2b32-4418-a05b-da30146a85a0-routing-overlay-volume" pod "ebc0fdde-2b32-4418-a05b-da30146a85a0" (UID: "ebc0fdde-2b32-4418-a05b-da30146a85a0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:49.158307 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.158292 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc0fdde-2b32-4418-a05b-da30146a85a0-plugins-config-volume" pod "ebc0fdde-2b32-4418-a05b-da30146a85a0" (UID: "ebc0fdde-2b32-4418-a05b-da30146a85a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:49.158524 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.158509 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc0fdde-2b32-4418-a05b-da30146a85a0-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:09:49.158562 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.158527 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ebc0fdde-2b32-4418-a05b-da30146a85a0-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:09:49.160285 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.160265 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc0fdde-2b32-4418-a05b-da30146a85a0-provider-credentials" pod "ebc0fdde-2b32-4418-a05b-da30146a85a0" (UID: "ebc0fdde-2b32-4418-a05b-da30146a85a0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:49.160601 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.160581 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc0fdde-2b32-4418-a05b-da30146a85a0-kube-api-access-8lpzx" pod "ebc0fdde-2b32-4418-a05b-da30146a85a0" (UID: "ebc0fdde-2b32-4418-a05b-da30146a85a0"). InnerVolumeSpecName "kube-api-access-8lpzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:49.259484 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.259454 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ebc0fdde-2b32-4418-a05b-da30146a85a0-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:09:49.259484 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.259481 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lpzx\" (UniqueName: \"kubernetes.io/projected/ebc0fdde-2b32-4418-a05b-da30146a85a0-kube-api-access-8lpzx\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:09:49.873915 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.873879 4861 generic.go:356] "Generic (PLEG): container finished" podID="ebc0fdde-2b32-4418-a05b-da30146a85a0" containerID="e322154633887ee4da17616db2d3bf8ee964bc65303d63696b4f36f915f1f156" exitCode=0 Oct 30 17:09:49.874383 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.873929 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" Oct 30 17:09:49.874383 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.873931 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" event={"ID":"ebc0fdde-2b32-4418-a05b-da30146a85a0","Type":"ContainerDied","Data":"e322154633887ee4da17616db2d3bf8ee964bc65303d63696b4f36f915f1f156"} Oct 30 17:09:49.874383 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.874037 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb" event={"ID":"ebc0fdde-2b32-4418-a05b-da30146a85a0","Type":"ContainerDied","Data":"7163ad0f64428dd7ee0d94cdecfe5f02fe057302917143729490749e40d5cf5f"} Oct 30 17:09:49.874383 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.874060 4861 scope.go:122] "RemoveContainer" containerID="e322154633887ee4da17616db2d3bf8ee964bc65303d63696b4f36f915f1f156" Oct 30 17:09:49.888028 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.888011 4861 scope.go:122] "RemoveContainer" containerID="e322154633887ee4da17616db2d3bf8ee964bc65303d63696b4f36f915f1f156" Oct 30 17:09:49.888307 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:49.888289 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e322154633887ee4da17616db2d3bf8ee964bc65303d63696b4f36f915f1f156\": container with ID starting with e322154633887ee4da17616db2d3bf8ee964bc65303d63696b4f36f915f1f156 not found: ID does not exist" containerID="e322154633887ee4da17616db2d3bf8ee964bc65303d63696b4f36f915f1f156" Oct 30 17:09:49.888361 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.888314 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e322154633887ee4da17616db2d3bf8ee964bc65303d63696b4f36f915f1f156"} err="failed to get container status \"e322154633887ee4da17616db2d3bf8ee964bc65303d63696b4f36f915f1f156\": rpc error: code = NotFound desc = could not find container \"e322154633887ee4da17616db2d3bf8ee964bc65303d63696b4f36f915f1f156\": container with ID starting with e322154633887ee4da17616db2d3bf8ee964bc65303d63696b4f36f915f1f156 not found: ID does not exist" Oct 30 17:09:49.932406 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.932381 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb"] Oct 30 17:09:49.946098 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:49.946074 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-566877xxb"] Oct 30 17:09:50.290249 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:50.290217 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:09:51.811799 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:51.811704 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebc0fdde-2b32-4418-a05b-da30146a85a0" path="/var/lib/kubelet/pods/ebc0fdde-2b32-4418-a05b-da30146a85a0/volumes" Oct 30 17:09:57.643200 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.643163 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg"] Oct 30 17:09:57.644072 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.644041 4861 cpu_manager.go:414] "containerMap: removing container" podUID="ebc0fdde-2b32-4418-a05b-da30146a85a0" containerName="payload-processing" Oct 30 17:09:57.644072 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.644071 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc0fdde-2b32-4418-a05b-da30146a85a0" containerName="payload-processing" Oct 30 17:09:57.644282 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.644201 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc0fdde-2b32-4418-a05b-da30146a85a0" containerName="payload-processing" Oct 30 17:09:57.649940 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.649897 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:09:57.667530 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.667502 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg"] Oct 30 17:09:57.677366 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.677336 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"payload-processing-external-model-e2e--8e3f1132-dockercfg-4lq89\"" Oct 30 17:09:57.677490 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.677374 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2589b5\"" Oct 30 17:09:57.677490 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.677418 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"agc-cred-c1ddc76883fe\"" Oct 30 17:09:57.677490 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.677454 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"openshift-service-ca.crt\"" Oct 30 17:09:57.677654 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.677528 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"routing-overlay\"" Oct 30 17:09:57.682879 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.682861 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"kube-root-ca.crt\"" Oct 30 17:09:57.822751 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.818700 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwsvl\" (UniqueName: \"kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-kube-api-access-hwsvl\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg\" (UID: \"dbd7fa86-4585-4298-940e-b3838f3064a1\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:09:57.822751 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.818957 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd7fa86-4585-4298-940e-b3838f3064a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg\" (UID: \"dbd7fa86-4585-4298-940e-b3838f3064a1\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:09:57.822751 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.819015 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg\" (UID: \"dbd7fa86-4585-4298-940e-b3838f3064a1\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:09:57.822751 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.819150 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd7fa86-4585-4298-940e-b3838f3064a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg\" (UID: \"dbd7fa86-4585-4298-940e-b3838f3064a1\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:09:57.847756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.847708 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w"] Oct 30 17:09:57.920332 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.920254 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsvl\" (UniqueName: \"kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-kube-api-access-hwsvl\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg\" (UID: \"dbd7fa86-4585-4298-940e-b3838f3064a1\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:09:57.920502 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.920376 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd7fa86-4585-4298-940e-b3838f3064a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg\" (UID: \"dbd7fa86-4585-4298-940e-b3838f3064a1\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:09:57.920502 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.920398 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg\" (UID: \"dbd7fa86-4585-4298-940e-b3838f3064a1\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:09:57.920502 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.920456 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd7fa86-4585-4298-940e-b3838f3064a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg\" (UID: \"dbd7fa86-4585-4298-940e-b3838f3064a1\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:09:57.920821 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:57.920590 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2589b5/agc-cred-c1ddc76883fe: secret "agc-cred-c1ddc76883fe" not found Oct 30 17:09:57.920821 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:57.920609 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg: secret "agc-cred-c1ddc76883fe" not found Oct 30 17:09:57.920821 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:57.920664 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-provider-credentials podName:dbd7fa86-4585-4298-940e-b3838f3064a1 nodeName:}" failed. No retries permitted until 2026-10-30 17:09:58.420643591 +0000 UTC m=+2828.763317611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" (UID: "dbd7fa86-4585-4298-940e-b3838f3064a1") : secret "agc-cred-c1ddc76883fe" not found Oct 30 17:09:57.921203 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.921180 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd7fa86-4585-4298-940e-b3838f3064a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg\" (UID: \"dbd7fa86-4585-4298-940e-b3838f3064a1\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:09:57.921293 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.921215 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd7fa86-4585-4298-940e-b3838f3064a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg\" (UID: \"dbd7fa86-4585-4298-940e-b3838f3064a1\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:09:57.947790 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:57.947769 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwsvl\" (UniqueName: \"kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-kube-api-access-hwsvl\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg\" (UID: \"dbd7fa86-4585-4298-940e-b3838f3064a1\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:09:58.426559 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:58.426515 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg\" (UID: \"dbd7fa86-4585-4298-940e-b3838f3064a1\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:09:58.426804 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:58.426658 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2589b5/agc-cred-c1ddc76883fe: secret "agc-cred-c1ddc76883fe" not found Oct 30 17:09:58.426804 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:58.426678 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg: secret "agc-cred-c1ddc76883fe" not found Oct 30 17:09:58.426804 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:58.426774 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-provider-credentials podName:dbd7fa86-4585-4298-940e-b3838f3064a1 nodeName:}" failed. No retries permitted until 2026-10-30 17:09:59.426756377 +0000 UTC m=+2829.769430407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" (UID: "dbd7fa86-4585-4298-940e-b3838f3064a1") : secret "agc-cred-c1ddc76883fe" not found Oct 30 17:09:58.714322 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:58.714222 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg"] Oct 30 17:09:58.714804 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:58.714669 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" podUID="dbd7fa86-4585-4298-940e-b3838f3064a1" Oct 30 17:09:58.988550 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:58.988505 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:09:58.998007 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:58.997979 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:09:59.138152 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:59.138122 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd7fa86-4585-4298-940e-b3838f3064a1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd7fa86-4585-4298-940e-b3838f3064a1-routing-overlay-volume\") pod \"dbd7fa86-4585-4298-940e-b3838f3064a1\" (UID: \"dbd7fa86-4585-4298-940e-b3838f3064a1\") " Oct 30 17:09:59.138376 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:59.138170 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd7fa86-4585-4298-940e-b3838f3064a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd7fa86-4585-4298-940e-b3838f3064a1-plugins-config-volume\") pod \"dbd7fa86-4585-4298-940e-b3838f3064a1\" (UID: \"dbd7fa86-4585-4298-940e-b3838f3064a1\") " Oct 30 17:09:59.138376 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:59.138331 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-kube-api-access-hwsvl\" (UniqueName: \"kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-kube-api-access-hwsvl\") pod \"dbd7fa86-4585-4298-940e-b3838f3064a1\" (UID: \"dbd7fa86-4585-4298-940e-b3838f3064a1\") " Oct 30 17:09:59.138580 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:59.138555 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd7fa86-4585-4298-940e-b3838f3064a1-routing-overlay-volume" pod "dbd7fa86-4585-4298-940e-b3838f3064a1" (UID: "dbd7fa86-4585-4298-940e-b3838f3064a1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:59.138644 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:59.138628 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd7fa86-4585-4298-940e-b3838f3064a1-plugins-config-volume" pod "dbd7fa86-4585-4298-940e-b3838f3064a1" (UID: "dbd7fa86-4585-4298-940e-b3838f3064a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:09:59.139112 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:59.139086 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd7fa86-4585-4298-940e-b3838f3064a1-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:09:59.139260 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:59.139115 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd7fa86-4585-4298-940e-b3838f3064a1-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:09:59.140706 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:59.140677 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-kube-api-access-hwsvl" pod "dbd7fa86-4585-4298-940e-b3838f3064a1" (UID: "dbd7fa86-4585-4298-940e-b3838f3064a1"). InnerVolumeSpecName "kube-api-access-hwsvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:59.240977 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:59.240908 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwsvl\" (UniqueName: \"kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-kube-api-access-hwsvl\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:09:59.444390 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:59.444356 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg\" (UID: \"dbd7fa86-4585-4298-940e-b3838f3064a1\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:09:59.444571 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:59.444502 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2589b5/agc-cred-c1ddc76883fe: secret "agc-cred-c1ddc76883fe" not found Oct 30 17:09:59.444571 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:59.444522 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg: secret "agc-cred-c1ddc76883fe" not found Oct 30 17:09:59.444571 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:09:59.444572 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-provider-credentials podName:dbd7fa86-4585-4298-940e-b3838f3064a1 nodeName:}" failed. No retries permitted until 2026-10-30 17:10:01.444557423 +0000 UTC m=+2831.787231457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" (UID: "dbd7fa86-4585-4298-940e-b3838f3064a1") : secret "agc-cred-c1ddc76883fe" not found Oct 30 17:09:59.995554 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:09:59.995525 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg" Oct 30 17:10:00.084388 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:00.084310 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg"] Oct 30 17:10:00.088506 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:00.088471 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-7bbb6w4dg"] Oct 30 17:10:00.254495 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:00.254449 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd7fa86-4585-4298-940e-b3838f3064a1-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:01.809699 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:01.809653 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd7fa86-4585-4298-940e-b3838f3064a1" path="/var/lib/kubelet/pods/dbd7fa86-4585-4298-940e-b3838f3064a1/volumes" Oct 30 17:10:02.848595 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:02.848549 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" podUID="9971fa48-be67-4483-9584-3e96b8bfb84c" containerName="payload-processing" containerID="cri-o://3fafef9bb3a3aea542a3aaf0dd4cbddc1647c2780fe4fff255a9df60d45e2154" gracePeriod=25 Oct 30 17:10:03.051870 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:03.051832 4861 generic.go:356] "Generic (PLEG): container finished" podID="9971fa48-be67-4483-9584-3e96b8bfb84c" containerID="3fafef9bb3a3aea542a3aaf0dd4cbddc1647c2780fe4fff255a9df60d45e2154" exitCode=0 Oct 30 17:10:03.052058 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:03.051897 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" event={"ID":"9971fa48-be67-4483-9584-3e96b8bfb84c","Type":"ContainerDied","Data":"3fafef9bb3a3aea542a3aaf0dd4cbddc1647c2780fe4fff255a9df60d45e2154"} Oct 30 17:10:03.144515 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:03.144486 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:10:03.204004 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:03.203974 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9971fa48-be67-4483-9584-3e96b8bfb84c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9971fa48-be67-4483-9584-3e96b8bfb84c-provider-credentials\") pod \"9971fa48-be67-4483-9584-3e96b8bfb84c\" (UID: \"9971fa48-be67-4483-9584-3e96b8bfb84c\") " Oct 30 17:10:03.204200 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:03.204129 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9971fa48-be67-4483-9584-3e96b8bfb84c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9971fa48-be67-4483-9584-3e96b8bfb84c-plugins-config-volume\") pod \"9971fa48-be67-4483-9584-3e96b8bfb84c\" (UID: \"9971fa48-be67-4483-9584-3e96b8bfb84c\") " Oct 30 17:10:03.204375 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:03.204356 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9971fa48-be67-4483-9584-3e96b8bfb84c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9971fa48-be67-4483-9584-3e96b8bfb84c-routing-overlay-volume\") pod \"9971fa48-be67-4483-9584-3e96b8bfb84c\" (UID: \"9971fa48-be67-4483-9584-3e96b8bfb84c\") " Oct 30 17:10:03.204469 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:03.204399 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9971fa48-be67-4483-9584-3e96b8bfb84c-kube-api-access-2mqhf\" (UniqueName: \"kubernetes.io/projected/9971fa48-be67-4483-9584-3e96b8bfb84c-kube-api-access-2mqhf\") pod \"9971fa48-be67-4483-9584-3e96b8bfb84c\" (UID: \"9971fa48-be67-4483-9584-3e96b8bfb84c\") " Oct 30 17:10:03.205903 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:03.205871 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9971fa48-be67-4483-9584-3e96b8bfb84c-plugins-config-volume" pod "9971fa48-be67-4483-9584-3e96b8bfb84c" (UID: "9971fa48-be67-4483-9584-3e96b8bfb84c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:03.206059 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:03.206038 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9971fa48-be67-4483-9584-3e96b8bfb84c-routing-overlay-volume" pod "9971fa48-be67-4483-9584-3e96b8bfb84c" (UID: "9971fa48-be67-4483-9584-3e96b8bfb84c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:03.227208 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:03.227178 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9971fa48-be67-4483-9584-3e96b8bfb84c-provider-credentials" pod "9971fa48-be67-4483-9584-3e96b8bfb84c" (UID: "9971fa48-be67-4483-9584-3e96b8bfb84c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:03.227348 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:03.227295 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9971fa48-be67-4483-9584-3e96b8bfb84c-kube-api-access-2mqhf" pod "9971fa48-be67-4483-9584-3e96b8bfb84c" (UID: "9971fa48-be67-4483-9584-3e96b8bfb84c"). InnerVolumeSpecName "kube-api-access-2mqhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:03.306522 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:03.306489 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9971fa48-be67-4483-9584-3e96b8bfb84c-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:03.306522 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:03.306520 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mqhf\" (UniqueName: \"kubernetes.io/projected/9971fa48-be67-4483-9584-3e96b8bfb84c-kube-api-access-2mqhf\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:03.306522 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:03.306530 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9971fa48-be67-4483-9584-3e96b8bfb84c-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:03.306811 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:03.306540 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9971fa48-be67-4483-9584-3e96b8bfb84c-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:04.061407 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:04.061369 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" event={"ID":"9971fa48-be67-4483-9584-3e96b8bfb84c","Type":"ContainerDied","Data":"8d028c4a5d4250af41aa7f52be2602dfc97e3642ff1572941329dc5aa7f3eeef"} Oct 30 17:10:04.061873 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:04.061437 4861 scope.go:122] "RemoveContainer" containerID="3fafef9bb3a3aea542a3aaf0dd4cbddc1647c2780fe4fff255a9df60d45e2154" Oct 30 17:10:04.061873 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:04.061448 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w" Oct 30 17:10:04.147323 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:04.147239 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w"] Oct 30 17:10:04.166312 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:04.166284 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6586dw59w"] Oct 30 17:10:05.811856 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:05.811812 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9971fa48-be67-4483-9584-3e96b8bfb84c" path="/var/lib/kubelet/pods/9971fa48-be67-4483-9584-3e96b8bfb84c/volumes" Oct 30 17:10:15.556812 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.556777 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k"] Oct 30 17:10:15.557664 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.557642 4861 cpu_manager.go:414] "containerMap: removing container" podUID="9971fa48-be67-4483-9584-3e96b8bfb84c" containerName="payload-processing" Oct 30 17:10:15.557802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.557665 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9971fa48-be67-4483-9584-3e96b8bfb84c" containerName="payload-processing" Oct 30 17:10:15.557896 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.557880 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="9971fa48-be67-4483-9584-3e96b8bfb84c" containerName="payload-processing" Oct 30 17:10:15.563033 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.563013 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:15.582204 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.582180 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k"] Oct 30 17:10:15.591855 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.591833 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3b3debd510cb\"" Oct 30 17:10:15.677160 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.677134 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgvqs\" (UniqueName: \"kubernetes.io/projected/12996239-1c6d-4700-a888-3939204690f7-kube-api-access-xgvqs\") pod \"payload-processing-external-model-64f4dc697d-kb64k\" (UID: \"12996239-1c6d-4700-a888-3939204690f7\") " pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:15.677303 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.677264 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12996239-1c6d-4700-a888-3939204690f7-plugins-config-volume\") pod \"payload-processing-external-model-64f4dc697d-kb64k\" (UID: \"12996239-1c6d-4700-a888-3939204690f7\") " pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:15.677303 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.677291 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12996239-1c6d-4700-a888-3939204690f7-provider-credentials\") pod \"payload-processing-external-model-64f4dc697d-kb64k\" (UID: \"12996239-1c6d-4700-a888-3939204690f7\") " pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:15.677504 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.677482 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12996239-1c6d-4700-a888-3939204690f7-routing-overlay-volume\") pod \"payload-processing-external-model-64f4dc697d-kb64k\" (UID: \"12996239-1c6d-4700-a888-3939204690f7\") " pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:15.778509 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.778484 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12996239-1c6d-4700-a888-3939204690f7-routing-overlay-volume\") pod \"payload-processing-external-model-64f4dc697d-kb64k\" (UID: \"12996239-1c6d-4700-a888-3939204690f7\") " pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:15.778630 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.778546 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvqs\" (UniqueName: \"kubernetes.io/projected/12996239-1c6d-4700-a888-3939204690f7-kube-api-access-xgvqs\") pod \"payload-processing-external-model-64f4dc697d-kb64k\" (UID: \"12996239-1c6d-4700-a888-3939204690f7\") " pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:15.778679 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.778647 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12996239-1c6d-4700-a888-3939204690f7-plugins-config-volume\") pod \"payload-processing-external-model-64f4dc697d-kb64k\" (UID: \"12996239-1c6d-4700-a888-3939204690f7\") " pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:15.778679 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.778674 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12996239-1c6d-4700-a888-3939204690f7-provider-credentials\") pod \"payload-processing-external-model-64f4dc697d-kb64k\" (UID: \"12996239-1c6d-4700-a888-3939204690f7\") " pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:15.779263 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.779236 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12996239-1c6d-4700-a888-3939204690f7-routing-overlay-volume\") pod \"payload-processing-external-model-64f4dc697d-kb64k\" (UID: \"12996239-1c6d-4700-a888-3939204690f7\") " pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:15.779495 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.779476 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12996239-1c6d-4700-a888-3939204690f7-plugins-config-volume\") pod \"payload-processing-external-model-64f4dc697d-kb64k\" (UID: \"12996239-1c6d-4700-a888-3939204690f7\") " pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:15.781474 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.781455 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12996239-1c6d-4700-a888-3939204690f7-provider-credentials\") pod \"payload-processing-external-model-64f4dc697d-kb64k\" (UID: \"12996239-1c6d-4700-a888-3939204690f7\") " pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:15.810767 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.810679 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvqs\" (UniqueName: \"kubernetes.io/projected/12996239-1c6d-4700-a888-3939204690f7-kube-api-access-xgvqs\") pod \"payload-processing-external-model-64f4dc697d-kb64k\" (UID: \"12996239-1c6d-4700-a888-3939204690f7\") " pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:15.878901 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:15.878873 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:16.192275 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:16.192250 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k"] Oct 30 17:10:16.193217 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:10:16.193192 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12996239_1c6d_4700_a888_3939204690f7.slice/crio-d86455ce2230612c61d93089773e4df64ceaee7498a6ce041d17f1787540f58d WatchSource:0}: Error finding container d86455ce2230612c61d93089773e4df64ceaee7498a6ce041d17f1787540f58d: Status 404 returned error can't find the container with id d86455ce2230612c61d93089773e4df64ceaee7498a6ce041d17f1787540f58d Oct 30 17:10:17.171339 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:17.171285 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" event={"ID":"12996239-1c6d-4700-a888-3939204690f7","Type":"ContainerStarted","Data":"001b755ac773db48ddd223c3a9c5ee1e485846a7b02f00924334bb55bbdefe06"} Oct 30 17:10:17.171898 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:17.171353 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" event={"ID":"12996239-1c6d-4700-a888-3939204690f7","Type":"ContainerStarted","Data":"d86455ce2230612c61d93089773e4df64ceaee7498a6ce041d17f1787540f58d"} Oct 30 17:10:17.171898 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:17.171396 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:17.218651 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:17.218600 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" podStartSLOduration=2.218585239 podStartE2EDuration="2.218585239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:17.218053949 +0000 UTC m=+2847.560727994" watchObservedRunningTime="2026-10-30 17:10:17.218585239 +0000 UTC m=+2847.561259277" Oct 30 17:10:18.297716 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:18.297679 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k"] Oct 30 17:10:20.233367 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.233333 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m"] Oct 30 17:10:20.238374 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.238347 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:20.258272 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.258247 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m"] Oct 30 17:10:20.267474 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.267446 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1f349073ea4c\"" Oct 30 17:10:20.293248 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:10:20.293212 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:10:20.339437 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.339395 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52f3a8b2-c05a-49d5-bd33-9822571ebf05-routing-overlay-volume\") pod \"payload-processing-external-model-75df6f9dc7-2gx8m\" (UID: \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:20.339645 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.339451 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52f3a8b2-c05a-49d5-bd33-9822571ebf05-plugins-config-volume\") pod \"payload-processing-external-model-75df6f9dc7-2gx8m\" (UID: \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:20.339806 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.339783 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x676f\" (UniqueName: \"kubernetes.io/projected/52f3a8b2-c05a-49d5-bd33-9822571ebf05-kube-api-access-x676f\") pod \"payload-processing-external-model-75df6f9dc7-2gx8m\" (UID: \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:20.339892 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.339837 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52f3a8b2-c05a-49d5-bd33-9822571ebf05-provider-credentials\") pod \"payload-processing-external-model-75df6f9dc7-2gx8m\" (UID: \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:20.440652 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.440620 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52f3a8b2-c05a-49d5-bd33-9822571ebf05-provider-credentials\") pod \"payload-processing-external-model-75df6f9dc7-2gx8m\" (UID: \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:20.440854 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.440754 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52f3a8b2-c05a-49d5-bd33-9822571ebf05-routing-overlay-volume\") pod \"payload-processing-external-model-75df6f9dc7-2gx8m\" (UID: \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:20.440854 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.440780 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52f3a8b2-c05a-49d5-bd33-9822571ebf05-plugins-config-volume\") pod \"payload-processing-external-model-75df6f9dc7-2gx8m\" (UID: \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:20.440991 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.440920 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x676f\" (UniqueName: \"kubernetes.io/projected/52f3a8b2-c05a-49d5-bd33-9822571ebf05-kube-api-access-x676f\") pod \"payload-processing-external-model-75df6f9dc7-2gx8m\" (UID: \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:20.441519 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.441492 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52f3a8b2-c05a-49d5-bd33-9822571ebf05-routing-overlay-volume\") pod \"payload-processing-external-model-75df6f9dc7-2gx8m\" (UID: \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:20.441621 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.441559 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52f3a8b2-c05a-49d5-bd33-9822571ebf05-plugins-config-volume\") pod \"payload-processing-external-model-75df6f9dc7-2gx8m\" (UID: \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:20.443411 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.443387 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52f3a8b2-c05a-49d5-bd33-9822571ebf05-provider-credentials\") pod \"payload-processing-external-model-75df6f9dc7-2gx8m\" (UID: \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:20.467181 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.467153 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x676f\" (UniqueName: \"kubernetes.io/projected/52f3a8b2-c05a-49d5-bd33-9822571ebf05-kube-api-access-x676f\") pod \"payload-processing-external-model-75df6f9dc7-2gx8m\" (UID: \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\") " pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:20.554833 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.554756 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:20.926877 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:20.926851 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m"] Oct 30 17:10:20.927182 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:10:20.927159 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f3a8b2_c05a_49d5_bd33_9822571ebf05.slice/crio-a51f2531493614746a7e40ed2133eb32183739481f2587d50af784893680c525 WatchSource:0}: Error finding container a51f2531493614746a7e40ed2133eb32183739481f2587d50af784893680c525: Status 404 returned error can't find the container with id a51f2531493614746a7e40ed2133eb32183739481f2587d50af784893680c525 Oct 30 17:10:21.202337 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:21.202245 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" event={"ID":"52f3a8b2-c05a-49d5-bd33-9822571ebf05","Type":"ContainerStarted","Data":"0b3623a996356915928f3fc32b3e79561d2be07b1ad47433ac2f4daec2aa9f1f"} Oct 30 17:10:21.202337 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:21.202303 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" event={"ID":"52f3a8b2-c05a-49d5-bd33-9822571ebf05","Type":"ContainerStarted","Data":"a51f2531493614746a7e40ed2133eb32183739481f2587d50af784893680c525"} Oct 30 17:10:21.202337 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:21.202323 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:21.250295 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:21.250245 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" podStartSLOduration=1.250232188 podStartE2EDuration="1.250232188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:21.24626678 +0000 UTC m=+2851.588940814" watchObservedRunningTime="2026-10-30 17:10:21.250232188 +0000 UTC m=+2851.592906264" Oct 30 17:10:23.958604 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:23.958566 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r"] Oct 30 17:10:23.974506 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:23.974470 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m"] Oct 30 17:10:24.186495 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:24.186448 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" podUID="12996239-1c6d-4700-a888-3939204690f7" containerName="payload-processing" containerID="cri-o://001b755ac773db48ddd223c3a9c5ee1e485846a7b02f00924334bb55bbdefe06" gracePeriod=25 Oct 30 17:10:24.535348 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:24.535315 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:24.703069 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:24.703040 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12996239-1c6d-4700-a888-3939204690f7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/12996239-1c6d-4700-a888-3939204690f7-provider-credentials\") pod \"12996239-1c6d-4700-a888-3939204690f7\" (UID: \"12996239-1c6d-4700-a888-3939204690f7\") " Oct 30 17:10:24.703259 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:24.703096 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12996239-1c6d-4700-a888-3939204690f7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12996239-1c6d-4700-a888-3939204690f7-routing-overlay-volume\") pod \"12996239-1c6d-4700-a888-3939204690f7\" (UID: \"12996239-1c6d-4700-a888-3939204690f7\") " Oct 30 17:10:24.703259 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:24.703141 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12996239-1c6d-4700-a888-3939204690f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12996239-1c6d-4700-a888-3939204690f7-plugins-config-volume\") pod \"12996239-1c6d-4700-a888-3939204690f7\" (UID: \"12996239-1c6d-4700-a888-3939204690f7\") " Oct 30 17:10:24.703395 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:24.703261 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12996239-1c6d-4700-a888-3939204690f7-kube-api-access-xgvqs\" (UniqueName: \"kubernetes.io/projected/12996239-1c6d-4700-a888-3939204690f7-kube-api-access-xgvqs\") pod \"12996239-1c6d-4700-a888-3939204690f7\" (UID: \"12996239-1c6d-4700-a888-3939204690f7\") " Oct 30 17:10:24.703567 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:24.703542 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12996239-1c6d-4700-a888-3939204690f7-routing-overlay-volume" pod "12996239-1c6d-4700-a888-3939204690f7" (UID: "12996239-1c6d-4700-a888-3939204690f7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:24.703640 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:24.703604 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12996239-1c6d-4700-a888-3939204690f7-plugins-config-volume" pod "12996239-1c6d-4700-a888-3939204690f7" (UID: "12996239-1c6d-4700-a888-3939204690f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:24.704055 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:24.703934 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/12996239-1c6d-4700-a888-3939204690f7-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.704055 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:24.703960 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12996239-1c6d-4700-a888-3939204690f7-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.705660 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:24.705636 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12996239-1c6d-4700-a888-3939204690f7-provider-credentials" pod "12996239-1c6d-4700-a888-3939204690f7" (UID: "12996239-1c6d-4700-a888-3939204690f7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:24.706059 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:24.706022 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12996239-1c6d-4700-a888-3939204690f7-kube-api-access-xgvqs" pod "12996239-1c6d-4700-a888-3939204690f7" (UID: "12996239-1c6d-4700-a888-3939204690f7"). InnerVolumeSpecName "kube-api-access-xgvqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:24.805095 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:24.805031 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgvqs\" (UniqueName: \"kubernetes.io/projected/12996239-1c6d-4700-a888-3939204690f7-kube-api-access-xgvqs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.805095 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:24.805058 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/12996239-1c6d-4700-a888-3939204690f7-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:25.232533 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.232504 4861 generic.go:356] "Generic (PLEG): container finished" podID="12996239-1c6d-4700-a888-3939204690f7" containerID="001b755ac773db48ddd223c3a9c5ee1e485846a7b02f00924334bb55bbdefe06" exitCode=0 Oct 30 17:10:25.233007 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.232553 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" Oct 30 17:10:25.233007 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.232581 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" event={"ID":"12996239-1c6d-4700-a888-3939204690f7","Type":"ContainerDied","Data":"001b755ac773db48ddd223c3a9c5ee1e485846a7b02f00924334bb55bbdefe06"} Oct 30 17:10:25.233007 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.232638 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k" event={"ID":"12996239-1c6d-4700-a888-3939204690f7","Type":"ContainerDied","Data":"d86455ce2230612c61d93089773e4df64ceaee7498a6ce041d17f1787540f58d"} Oct 30 17:10:25.233007 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.232668 4861 scope.go:122] "RemoveContainer" containerID="001b755ac773db48ddd223c3a9c5ee1e485846a7b02f00924334bb55bbdefe06" Oct 30 17:10:25.248141 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.248120 4861 scope.go:122] "RemoveContainer" containerID="001b755ac773db48ddd223c3a9c5ee1e485846a7b02f00924334bb55bbdefe06" Oct 30 17:10:25.248443 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:10:25.248422 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"001b755ac773db48ddd223c3a9c5ee1e485846a7b02f00924334bb55bbdefe06\": container with ID starting with 001b755ac773db48ddd223c3a9c5ee1e485846a7b02f00924334bb55bbdefe06 not found: ID does not exist" containerID="001b755ac773db48ddd223c3a9c5ee1e485846a7b02f00924334bb55bbdefe06" Oct 30 17:10:25.248496 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.248452 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001b755ac773db48ddd223c3a9c5ee1e485846a7b02f00924334bb55bbdefe06"} err="failed to get container status \"001b755ac773db48ddd223c3a9c5ee1e485846a7b02f00924334bb55bbdefe06\": rpc error: code = NotFound desc = could not find container \"001b755ac773db48ddd223c3a9c5ee1e485846a7b02f00924334bb55bbdefe06\": container with ID starting with 001b755ac773db48ddd223c3a9c5ee1e485846a7b02f00924334bb55bbdefe06 not found: ID does not exist" Oct 30 17:10:25.299542 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.293599 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k"] Oct 30 17:10:25.307902 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.307873 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-64f4dc697d-kb64k"] Oct 30 17:10:25.487926 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.487850 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4"] Oct 30 17:10:25.488670 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.488656 4861 cpu_manager.go:414] "containerMap: removing container" podUID="12996239-1c6d-4700-a888-3939204690f7" containerName="payload-processing" Oct 30 17:10:25.488737 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.488671 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12996239-1c6d-4700-a888-3939204690f7" containerName="payload-processing" Oct 30 17:10:25.488849 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.488839 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="12996239-1c6d-4700-a888-3939204690f7" containerName="payload-processing" Oct 30 17:10:25.494136 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.494113 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:10:25.511488 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.511463 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2589b5\"" Oct 30 17:10:25.512222 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.512202 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"agc-cred-b5a49548520a\"" Oct 30 17:10:25.512350 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.512280 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"payload-processing-external-model-e2e--8e3f1132-dockercfg-jqdh6\"" Oct 30 17:10:25.513285 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.513266 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4"] Oct 30 17:10:25.530106 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.530074 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978ef4f2-ef94-42ed-abcc-01f26e1e5215-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4\" (UID: \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:10:25.530226 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.530140 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/978ef4f2-ef94-42ed-abcc-01f26e1e5215-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4\" (UID: \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:10:25.530408 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.530385 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/978ef4f2-ef94-42ed-abcc-01f26e1e5215-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4\" (UID: \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:10:25.530487 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.530447 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm7zx\" (UniqueName: \"kubernetes.io/projected/978ef4f2-ef94-42ed-abcc-01f26e1e5215-kube-api-access-lm7zx\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4\" (UID: \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:10:25.627748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.627112 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"routing-overlay\"" Oct 30 17:10:25.627748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.627202 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"openshift-service-ca.crt\"" Oct 30 17:10:25.627748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.627288 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"kube-root-ca.crt\"" Oct 30 17:10:25.631639 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.631604 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/978ef4f2-ef94-42ed-abcc-01f26e1e5215-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4\" (UID: \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:10:25.631817 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.631671 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm7zx\" (UniqueName: \"kubernetes.io/projected/978ef4f2-ef94-42ed-abcc-01f26e1e5215-kube-api-access-lm7zx\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4\" (UID: \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:10:25.631817 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.631783 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978ef4f2-ef94-42ed-abcc-01f26e1e5215-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4\" (UID: \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:10:25.631917 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.631827 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/978ef4f2-ef94-42ed-abcc-01f26e1e5215-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4\" (UID: \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:10:25.633341 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.633308 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/978ef4f2-ef94-42ed-abcc-01f26e1e5215-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4\" (UID: \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:10:25.633473 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.633455 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978ef4f2-ef94-42ed-abcc-01f26e1e5215-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4\" (UID: \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:10:25.644497 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.644466 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/978ef4f2-ef94-42ed-abcc-01f26e1e5215-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4\" (UID: \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:10:25.664482 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.664439 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm7zx\" (UniqueName: \"kubernetes.io/projected/978ef4f2-ef94-42ed-abcc-01f26e1e5215-kube-api-access-lm7zx\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4\" (UID: \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:10:25.809509 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.809426 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12996239-1c6d-4700-a888-3939204690f7" path="/var/lib/kubelet/pods/12996239-1c6d-4700-a888-3939204690f7/volumes" Oct 30 17:10:25.809710 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:25.809692 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:10:26.128582 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:26.128559 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4"] Oct 30 17:10:26.131502 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:10:26.131470 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod978ef4f2_ef94_42ed_abcc_01f26e1e5215.slice/crio-67ab508aafb53ce8cf589c3582aab5791b82ec9cb7335da4860223ad53298a80 WatchSource:0}: Error finding container 67ab508aafb53ce8cf589c3582aab5791b82ec9cb7335da4860223ad53298a80: Status 404 returned error can't find the container with id 67ab508aafb53ce8cf589c3582aab5791b82ec9cb7335da4860223ad53298a80 Oct 30 17:10:26.242880 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:26.242847 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" event={"ID":"978ef4f2-ef94-42ed-abcc-01f26e1e5215","Type":"ContainerStarted","Data":"b0690191ff842d3f33ea56db6c9515e4131adbc0b0038a2255bcca4849d7c3a9"} Oct 30 17:10:26.243245 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:26.242901 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" event={"ID":"978ef4f2-ef94-42ed-abcc-01f26e1e5215","Type":"ContainerStarted","Data":"67ab508aafb53ce8cf589c3582aab5791b82ec9cb7335da4860223ad53298a80"} Oct 30 17:10:26.243245 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:26.243011 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:10:26.293095 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:26.293047 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" podStartSLOduration=1.2930344009999999 podStartE2EDuration="1.293034401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:26.289250713 +0000 UTC m=+2856.631924751" watchObservedRunningTime="2026-10-30 17:10:26.293034401 +0000 UTC m=+2856.635708471" Oct 30 17:10:28.362322 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:28.362289 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm"] Oct 30 17:10:28.959539 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:28.959503 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" podUID="b17d7937-c70a-49ae-ae78-8b277d1d39fb" containerName="payload-processing" containerID="cri-o://e6e682b1d701a0d51ddda0a359680784f9ea31c603bad6b38606fb7f7de2f879" gracePeriod=25 Oct 30 17:10:28.975213 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:28.975171 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" podUID="52f3a8b2-c05a-49d5-bd33-9822571ebf05" containerName="payload-processing" containerID="cri-o://0b3623a996356915928f3fc32b3e79561d2be07b1ad47433ac2f4daec2aa9f1f" gracePeriod=25 Oct 30 17:10:29.270890 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.270857 4861 generic.go:356] "Generic (PLEG): container finished" podID="52f3a8b2-c05a-49d5-bd33-9822571ebf05" containerID="0b3623a996356915928f3fc32b3e79561d2be07b1ad47433ac2f4daec2aa9f1f" exitCode=0 Oct 30 17:10:29.271145 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.271123 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" event={"ID":"52f3a8b2-c05a-49d5-bd33-9822571ebf05","Type":"ContainerDied","Data":"0b3623a996356915928f3fc32b3e79561d2be07b1ad47433ac2f4daec2aa9f1f"} Oct 30 17:10:29.272592 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.272569 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:10:29.272985 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.272958 4861 generic.go:356] "Generic (PLEG): container finished" podID="b17d7937-c70a-49ae-ae78-8b277d1d39fb" containerID="e6e682b1d701a0d51ddda0a359680784f9ea31c603bad6b38606fb7f7de2f879" exitCode=0 Oct 30 17:10:29.273086 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.273005 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" event={"ID":"b17d7937-c70a-49ae-ae78-8b277d1d39fb","Type":"ContainerDied","Data":"e6e682b1d701a0d51ddda0a359680784f9ea31c603bad6b38606fb7f7de2f879"} Oct 30 17:10:29.273086 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.273040 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" event={"ID":"b17d7937-c70a-49ae-ae78-8b277d1d39fb","Type":"ContainerDied","Data":"0d1a544bd2ff9bb38612c26cb8bc700b420c8b9a961b09814a897d263263491b"} Oct 30 17:10:29.273086 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.273060 4861 scope.go:122] "RemoveContainer" containerID="e6e682b1d701a0d51ddda0a359680784f9ea31c603bad6b38606fb7f7de2f879" Oct 30 17:10:29.300816 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.300795 4861 scope.go:122] "RemoveContainer" containerID="e6e682b1d701a0d51ddda0a359680784f9ea31c603bad6b38606fb7f7de2f879" Oct 30 17:10:29.301133 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:10:29.301112 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e682b1d701a0d51ddda0a359680784f9ea31c603bad6b38606fb7f7de2f879\": container with ID starting with e6e682b1d701a0d51ddda0a359680784f9ea31c603bad6b38606fb7f7de2f879 not found: ID does not exist" containerID="e6e682b1d701a0d51ddda0a359680784f9ea31c603bad6b38606fb7f7de2f879" Oct 30 17:10:29.301208 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.301141 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e682b1d701a0d51ddda0a359680784f9ea31c603bad6b38606fb7f7de2f879"} err="failed to get container status \"e6e682b1d701a0d51ddda0a359680784f9ea31c603bad6b38606fb7f7de2f879\": rpc error: code = NotFound desc = could not find container \"e6e682b1d701a0d51ddda0a359680784f9ea31c603bad6b38606fb7f7de2f879\": container with ID starting with e6e682b1d701a0d51ddda0a359680784f9ea31c603bad6b38606fb7f7de2f879 not found: ID does not exist" Oct 30 17:10:29.302122 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.302105 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:29.383357 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.383324 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b17d7937-c70a-49ae-ae78-8b277d1d39fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b17d7937-c70a-49ae-ae78-8b277d1d39fb-plugins-config-volume\") pod \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\" (UID: \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\") " Oct 30 17:10:29.383904 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.383373 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f3a8b2-c05a-49d5-bd33-9822571ebf05-provider-credentials\" (UniqueName: \"kubernetes.io/projected/52f3a8b2-c05a-49d5-bd33-9822571ebf05-provider-credentials\") pod \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\" (UID: \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\") " Oct 30 17:10:29.383904 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.383424 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b17d7937-c70a-49ae-ae78-8b277d1d39fb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b17d7937-c70a-49ae-ae78-8b277d1d39fb-routing-overlay-volume\") pod \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\" (UID: \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\") " Oct 30 17:10:29.383904 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.383496 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52f3a8b2-c05a-49d5-bd33-9822571ebf05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52f3a8b2-c05a-49d5-bd33-9822571ebf05-plugins-config-volume\") pod \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\" (UID: \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\") " Oct 30 17:10:29.383904 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.383574 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52f3a8b2-c05a-49d5-bd33-9822571ebf05-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52f3a8b2-c05a-49d5-bd33-9822571ebf05-routing-overlay-volume\") pod \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\" (UID: \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\") " Oct 30 17:10:29.383904 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.383877 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17d7937-c70a-49ae-ae78-8b277d1d39fb-plugins-config-volume" pod "b17d7937-c70a-49ae-ae78-8b277d1d39fb" (UID: "b17d7937-c70a-49ae-ae78-8b277d1d39fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:29.384203 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.383912 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17d7937-c70a-49ae-ae78-8b277d1d39fb-routing-overlay-volume" pod "b17d7937-c70a-49ae-ae78-8b277d1d39fb" (UID: "b17d7937-c70a-49ae-ae78-8b277d1d39fb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:29.384203 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.384041 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f3a8b2-c05a-49d5-bd33-9822571ebf05-plugins-config-volume" pod "52f3a8b2-c05a-49d5-bd33-9822571ebf05" (UID: "52f3a8b2-c05a-49d5-bd33-9822571ebf05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:29.384323 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.384276 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b17d7937-c70a-49ae-ae78-8b277d1d39fb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b17d7937-c70a-49ae-ae78-8b277d1d39fb-provider-credentials\") pod \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\" (UID: \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\") " Oct 30 17:10:29.384323 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.384318 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b17d7937-c70a-49ae-ae78-8b277d1d39fb-kube-api-access-5xvd4\" (UniqueName: \"kubernetes.io/projected/b17d7937-c70a-49ae-ae78-8b277d1d39fb-kube-api-access-5xvd4\") pod \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\" (UID: \"b17d7937-c70a-49ae-ae78-8b277d1d39fb\") " Oct 30 17:10:29.384443 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.384348 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f3a8b2-c05a-49d5-bd33-9822571ebf05-routing-overlay-volume" pod "52f3a8b2-c05a-49d5-bd33-9822571ebf05" (UID: "52f3a8b2-c05a-49d5-bd33-9822571ebf05"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:29.384443 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.384399 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f3a8b2-c05a-49d5-bd33-9822571ebf05-kube-api-access-x676f\" (UniqueName: \"kubernetes.io/projected/52f3a8b2-c05a-49d5-bd33-9822571ebf05-kube-api-access-x676f\") pod \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\" (UID: \"52f3a8b2-c05a-49d5-bd33-9822571ebf05\") " Oct 30 17:10:29.385004 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.384980 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b17d7937-c70a-49ae-ae78-8b277d1d39fb-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:29.385004 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.385006 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b17d7937-c70a-49ae-ae78-8b277d1d39fb-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:29.385211 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.385021 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52f3a8b2-c05a-49d5-bd33-9822571ebf05-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:29.385211 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.385048 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/52f3a8b2-c05a-49d5-bd33-9822571ebf05-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:29.386548 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.386473 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f3a8b2-c05a-49d5-bd33-9822571ebf05-provider-credentials" pod "52f3a8b2-c05a-49d5-bd33-9822571ebf05" (UID: "52f3a8b2-c05a-49d5-bd33-9822571ebf05"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:29.387420 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.386922 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17d7937-c70a-49ae-ae78-8b277d1d39fb-provider-credentials" pod "b17d7937-c70a-49ae-ae78-8b277d1d39fb" (UID: "b17d7937-c70a-49ae-ae78-8b277d1d39fb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:29.387525 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.387449 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17d7937-c70a-49ae-ae78-8b277d1d39fb-kube-api-access-5xvd4" pod "b17d7937-c70a-49ae-ae78-8b277d1d39fb" (UID: "b17d7937-c70a-49ae-ae78-8b277d1d39fb"). InnerVolumeSpecName "kube-api-access-5xvd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:29.387525 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.387473 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f3a8b2-c05a-49d5-bd33-9822571ebf05-kube-api-access-x676f" pod "52f3a8b2-c05a-49d5-bd33-9822571ebf05" (UID: "52f3a8b2-c05a-49d5-bd33-9822571ebf05"). InnerVolumeSpecName "kube-api-access-x676f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:29.486361 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.486275 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/52f3a8b2-c05a-49d5-bd33-9822571ebf05-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:29.486361 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.486305 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b17d7937-c70a-49ae-ae78-8b277d1d39fb-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:29.486361 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.486315 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xvd4\" (UniqueName: \"kubernetes.io/projected/b17d7937-c70a-49ae-ae78-8b277d1d39fb-kube-api-access-5xvd4\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:29.486361 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:29.486325 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x676f\" (UniqueName: \"kubernetes.io/projected/52f3a8b2-c05a-49d5-bd33-9822571ebf05-kube-api-access-x676f\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:30.281779 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:30.281743 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" Oct 30 17:10:30.281960 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:30.281752 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m" event={"ID":"52f3a8b2-c05a-49d5-bd33-9822571ebf05","Type":"ContainerDied","Data":"a51f2531493614746a7e40ed2133eb32183739481f2587d50af784893680c525"} Oct 30 17:10:30.281960 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:30.281882 4861 scope.go:122] "RemoveContainer" containerID="0b3623a996356915928f3fc32b3e79561d2be07b1ad47433ac2f4daec2aa9f1f" Oct 30 17:10:30.283285 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:30.283269 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r" Oct 30 17:10:30.344880 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:30.344847 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m"] Oct 30 17:10:30.358883 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:30.358849 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-75df6f9dc7-2gx8m"] Oct 30 17:10:30.389268 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:30.389237 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r"] Oct 30 17:10:30.398392 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:30.398360 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b4f7df67-twr2r"] Oct 30 17:10:31.810775 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:31.810704 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f3a8b2-c05a-49d5-bd33-9822571ebf05" path="/var/lib/kubelet/pods/52f3a8b2-c05a-49d5-bd33-9822571ebf05/volumes" Oct 30 17:10:31.811462 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:31.811440 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b17d7937-c70a-49ae-ae78-8b277d1d39fb" path="/var/lib/kubelet/pods/b17d7937-c70a-49ae-ae78-8b277d1d39fb/volumes" Oct 30 17:10:33.317240 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.317193 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv"] Oct 30 17:10:33.318101 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.318082 4861 cpu_manager.go:414] "containerMap: removing container" podUID="52f3a8b2-c05a-49d5-bd33-9822571ebf05" containerName="payload-processing" Oct 30 17:10:33.318177 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.318102 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f3a8b2-c05a-49d5-bd33-9822571ebf05" containerName="payload-processing" Oct 30 17:10:33.318177 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.318122 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b17d7937-c70a-49ae-ae78-8b277d1d39fb" containerName="payload-processing" Oct 30 17:10:33.318177 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.318127 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b17d7937-c70a-49ae-ae78-8b277d1d39fb" containerName="payload-processing" Oct 30 17:10:33.318318 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.318226 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f3a8b2-c05a-49d5-bd33-9822571ebf05" containerName="payload-processing" Oct 30 17:10:33.318318 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.318237 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b17d7937-c70a-49ae-ae78-8b277d1d39fb" containerName="payload-processing" Oct 30 17:10:33.323906 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.323885 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.348047 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.348023 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv"] Oct 30 17:10:33.373761 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.362193 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv"] Oct 30 17:10:33.373761 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.363535 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" podUID="746b39b4-7b4b-4fe3-b9eb-944af7954669" containerName="payload-processing" containerID="cri-o://e98144ed1e26608d6875bdf32de224578d32ee6c4c1b12a1d19ffaf9cb59e420" gracePeriod=25 Oct 30 17:10:33.373761 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.367678 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.385255 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.385227 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv"] Oct 30 17:10:33.451120 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.451092 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.451255 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.451198 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfj5k\" (UniqueName: \"kubernetes.io/projected/cd5c0df9-6522-46f6-8222-117b7ad17bea-kube-api-access-pfj5k\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.451255 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.451230 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.451385 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.451302 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.451438 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.451388 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adb920d-0538-46bb-8c96-3d5f68902e50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.451438 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.451429 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.451533 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.451448 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.451533 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.451485 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.451655 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.451603 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd5c0df9-6522-46f6-8222-117b7ad17bea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.451707 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.451664 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.451794 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.451705 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.451867 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.451845 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.451923 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.451890 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.452007 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.451989 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.452064 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.452038 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.452119 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.452059 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6pv5\" (UniqueName: \"kubernetes.io/projected/7adb920d-0538-46bb-8c96-3d5f68902e50-kube-api-access-h6pv5\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.452119 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.452079 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.452119 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.452099 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.503611 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:10:33.503581 4861 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746b39b4_7b4b_4fe3_b9eb_944af7954669.slice/crio-conmon-e98144ed1e26608d6875bdf32de224578d32ee6c4c1b12a1d19ffaf9cb59e420.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:10:33.553216 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553180 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.553413 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553232 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.553413 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553250 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6pv5\" (UniqueName: \"kubernetes.io/projected/7adb920d-0538-46bb-8c96-3d5f68902e50-kube-api-access-h6pv5\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.553549 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553413 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.553549 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553456 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.553549 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553506 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.553718 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553559 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfj5k\" (UniqueName: \"kubernetes.io/projected/cd5c0df9-6522-46f6-8222-117b7ad17bea-kube-api-access-pfj5k\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.553718 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553577 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.553718 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553608 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.553718 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553638 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adb920d-0538-46bb-8c96-3d5f68902e50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.553718 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553672 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.553718 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553679 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.553718 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553690 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.554189 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553796 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.554189 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553845 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd5c0df9-6522-46f6-8222-117b7ad17bea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.554189 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553913 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.554189 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553951 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.554189 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.553963 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.554189 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.554003 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.554189 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.554069 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.554189 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.554099 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.554682 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.554202 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-credential-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.554682 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.554248 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.554682 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.554536 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-workload-certs\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.554873 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.554796 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd5c0df9-6522-46f6-8222-117b7ad17bea-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.554873 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.554795 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adb920d-0538-46bb-8c96-3d5f68902e50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.554973 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.554950 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-data\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.555149 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.555133 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-workload-socket\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.556427 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.556401 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.556971 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.556944 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-envoy\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.557561 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.557537 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.557651 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.557539 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.585878 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.583630 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfj5k\" (UniqueName: \"kubernetes.io/projected/cd5c0df9-6522-46f6-8222-117b7ad17bea-kube-api-access-pfj5k\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.587536 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.587509 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-9s7vv\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.588713 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.588683 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6pv5\" (UniqueName: \"kubernetes.io/projected/7adb920d-0538-46bb-8c96-3d5f68902e50-kube-api-access-h6pv5\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.589615 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.589591 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-token\") pod \"maas-default-gateway-openshift-default-7854949c9-g2hzv\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.638917 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.638886 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:33.662745 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.662711 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:10:33.689017 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.688141 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:33.757109 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.757060 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746b39b4-7b4b-4fe3-b9eb-944af7954669-kube-api-access-dl67c\" (UniqueName: \"kubernetes.io/projected/746b39b4-7b4b-4fe3-b9eb-944af7954669-kube-api-access-dl67c\") pod \"746b39b4-7b4b-4fe3-b9eb-944af7954669\" (UID: \"746b39b4-7b4b-4fe3-b9eb-944af7954669\") " Oct 30 17:10:33.757313 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.757261 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746b39b4-7b4b-4fe3-b9eb-944af7954669-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/746b39b4-7b4b-4fe3-b9eb-944af7954669-routing-overlay-volume\") pod \"746b39b4-7b4b-4fe3-b9eb-944af7954669\" (UID: \"746b39b4-7b4b-4fe3-b9eb-944af7954669\") " Oct 30 17:10:33.757398 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.757385 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746b39b4-7b4b-4fe3-b9eb-944af7954669-provider-credentials\" (UniqueName: \"kubernetes.io/projected/746b39b4-7b4b-4fe3-b9eb-944af7954669-provider-credentials\") pod \"746b39b4-7b4b-4fe3-b9eb-944af7954669\" (UID: \"746b39b4-7b4b-4fe3-b9eb-944af7954669\") " Oct 30 17:10:33.757468 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.757431 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746b39b4-7b4b-4fe3-b9eb-944af7954669-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/746b39b4-7b4b-4fe3-b9eb-944af7954669-plugins-config-volume\") pod \"746b39b4-7b4b-4fe3-b9eb-944af7954669\" (UID: \"746b39b4-7b4b-4fe3-b9eb-944af7954669\") " Oct 30 17:10:33.758048 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.758021 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746b39b4-7b4b-4fe3-b9eb-944af7954669-routing-overlay-volume" pod "746b39b4-7b4b-4fe3-b9eb-944af7954669" (UID: "746b39b4-7b4b-4fe3-b9eb-944af7954669"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:33.758184 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.758145 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/746b39b4-7b4b-4fe3-b9eb-944af7954669-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:33.758508 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.758486 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746b39b4-7b4b-4fe3-b9eb-944af7954669-plugins-config-volume" pod "746b39b4-7b4b-4fe3-b9eb-944af7954669" (UID: "746b39b4-7b4b-4fe3-b9eb-944af7954669"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:33.762349 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.762323 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746b39b4-7b4b-4fe3-b9eb-944af7954669-provider-credentials" pod "746b39b4-7b4b-4fe3-b9eb-944af7954669" (UID: "746b39b4-7b4b-4fe3-b9eb-944af7954669"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:33.764247 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.764221 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746b39b4-7b4b-4fe3-b9eb-944af7954669-kube-api-access-dl67c" pod "746b39b4-7b4b-4fe3-b9eb-944af7954669" (UID: "746b39b4-7b4b-4fe3-b9eb-944af7954669"). InnerVolumeSpecName "kube-api-access-dl67c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:33.859783 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.859663 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/746b39b4-7b4b-4fe3-b9eb-944af7954669-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:33.859783 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.859700 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/746b39b4-7b4b-4fe3-b9eb-944af7954669-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:33.859783 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.859736 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl67c\" (UniqueName: \"kubernetes.io/projected/746b39b4-7b4b-4fe3-b9eb-944af7954669-kube-api-access-dl67c\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:10:33.975242 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:10:33.975205 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5c0df9_6522_46f6_8222_117b7ad17bea.slice/crio-7f4bf6b8561943c4655f3ea2c7bd67e7f3fa59ca4de8babda7305d4ea914e4b3 WatchSource:0}: Error finding container 7f4bf6b8561943c4655f3ea2c7bd67e7f3fa59ca4de8babda7305d4ea914e4b3: Status 404 returned error can't find the container with id 7f4bf6b8561943c4655f3ea2c7bd67e7f3fa59ca4de8babda7305d4ea914e4b3 Oct 30 17:10:33.975373 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.975262 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv"] Oct 30 17:10:33.978523 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.978490 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:10:33.978608 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.978557 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:10:33.978608 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:33.978585 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:10:34.041224 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.041194 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv"] Oct 30 17:10:34.041533 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:10:34.041505 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adb920d_0538_46bb_8c96_3d5f68902e50.slice/crio-671728be9011423659da0f24856205ef687d71a2f3c5ca163da75150cb2cfee0 WatchSource:0}: Error finding container 671728be9011423659da0f24856205ef687d71a2f3c5ca163da75150cb2cfee0: Status 404 returned error can't find the container with id 671728be9011423659da0f24856205ef687d71a2f3c5ca163da75150cb2cfee0 Oct 30 17:10:34.045138 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.045097 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:10:34.045250 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.045181 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:10:34.045622 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.045255 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:10:34.327512 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.325711 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" event={"ID":"cd5c0df9-6522-46f6-8222-117b7ad17bea","Type":"ContainerStarted","Data":"d4e4b0cfcc4b59396039dd445a7c8d4e67cf3e3feef5f0dd61c540b7946063ce"} Oct 30 17:10:34.327512 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.325805 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" event={"ID":"cd5c0df9-6522-46f6-8222-117b7ad17bea","Type":"ContainerStarted","Data":"7f4bf6b8561943c4655f3ea2c7bd67e7f3fa59ca4de8babda7305d4ea914e4b3"} Oct 30 17:10:34.330770 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.330708 4861 generic.go:356] "Generic (PLEG): container finished" podID="746b39b4-7b4b-4fe3-b9eb-944af7954669" containerID="e98144ed1e26608d6875bdf32de224578d32ee6c4c1b12a1d19ffaf9cb59e420" exitCode=0 Oct 30 17:10:34.330770 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.330753 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" event={"ID":"746b39b4-7b4b-4fe3-b9eb-944af7954669","Type":"ContainerDied","Data":"e98144ed1e26608d6875bdf32de224578d32ee6c4c1b12a1d19ffaf9cb59e420"} Oct 30 17:10:34.331024 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.330772 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" Oct 30 17:10:34.331024 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.330809 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm" event={"ID":"746b39b4-7b4b-4fe3-b9eb-944af7954669","Type":"ContainerDied","Data":"e4bd0ad133a8cfc65814e3fcb2c296753d6ffb0d772a3b22037b952f18905207"} Oct 30 17:10:34.331024 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.330831 4861 scope.go:122] "RemoveContainer" containerID="e98144ed1e26608d6875bdf32de224578d32ee6c4c1b12a1d19ffaf9cb59e420" Oct 30 17:10:34.334075 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.334048 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" event={"ID":"7adb920d-0538-46bb-8c96-3d5f68902e50","Type":"ContainerStarted","Data":"f9f009d6735de488925d6eb3767cc4919921a289738c60b1633effe2cb972bb4"} Oct 30 17:10:34.334185 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.334100 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" event={"ID":"7adb920d-0538-46bb-8c96-3d5f68902e50","Type":"ContainerStarted","Data":"671728be9011423659da0f24856205ef687d71a2f3c5ca163da75150cb2cfee0"} Oct 30 17:10:34.344802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.344782 4861 scope.go:122] "RemoveContainer" containerID="e98144ed1e26608d6875bdf32de224578d32ee6c4c1b12a1d19ffaf9cb59e420" Oct 30 17:10:34.345092 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:10:34.345069 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98144ed1e26608d6875bdf32de224578d32ee6c4c1b12a1d19ffaf9cb59e420\": container with ID starting with e98144ed1e26608d6875bdf32de224578d32ee6c4c1b12a1d19ffaf9cb59e420 not found: ID does not exist" containerID="e98144ed1e26608d6875bdf32de224578d32ee6c4c1b12a1d19ffaf9cb59e420" Oct 30 17:10:34.345220 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.345095 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98144ed1e26608d6875bdf32de224578d32ee6c4c1b12a1d19ffaf9cb59e420"} err="failed to get container status \"e98144ed1e26608d6875bdf32de224578d32ee6c4c1b12a1d19ffaf9cb59e420\": rpc error: code = NotFound desc = could not find container \"e98144ed1e26608d6875bdf32de224578d32ee6c4c1b12a1d19ffaf9cb59e420\": container with ID starting with e98144ed1e26608d6875bdf32de224578d32ee6c4c1b12a1d19ffaf9cb59e420 not found: ID does not exist" Oct 30 17:10:34.375061 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.374965 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" podStartSLOduration=1.3749476170000001 podStartE2EDuration="1.374947617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:34.373805397 +0000 UTC m=+2864.716479435" watchObservedRunningTime="2026-10-30 17:10:34.374947617 +0000 UTC m=+2864.717621656" Oct 30 17:10:34.414800 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.414774 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm"] Oct 30 17:10:34.418555 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.418528 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d5b798/payload-processing-external-model-e2e-extmod-a-d5b798-66d76ckvm"] Oct 30 17:10:34.464037 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.463980 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" podStartSLOduration=1.463966293 podStartE2EDuration="1.463966293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:34.46164385 +0000 UTC m=+2864.804317888" watchObservedRunningTime="2026-10-30 17:10:34.463966293 +0000 UTC m=+2864.806640330" Oct 30 17:10:34.639808 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.639716 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:34.688818 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:34.688776 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:35.643799 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:35.643405 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-9s7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:10:35.643799 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:35.643477 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" podUID="cd5c0df9-6522-46f6-8222-117b7ad17bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 17:10:35.691022 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:35.690977 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-g2hzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:10:35.691175 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:35.691061 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" podUID="7adb920d-0538-46bb-8c96-3d5f68902e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 17:10:35.855749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:35.838605 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="746b39b4-7b4b-4fe3-b9eb-944af7954669" path="/var/lib/kubelet/pods/746b39b4-7b4b-4fe3-b9eb-944af7954669/volumes" Oct 30 17:10:36.641546 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:36.641095 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-9s7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:10:36.641546 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:36.641157 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" podUID="cd5c0df9-6522-46f6-8222-117b7ad17bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 17:10:36.689321 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:36.689284 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-g2hzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:10:36.689746 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:36.689332 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" podUID="7adb920d-0538-46bb-8c96-3d5f68902e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 17:10:37.255747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:37.255655 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:10:37.643812 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:37.643242 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-9s7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:10:37.643812 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:37.643304 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" podUID="cd5c0df9-6522-46f6-8222-117b7ad17bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 17:10:37.692123 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:37.691163 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-g2hzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:10:37.692123 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:37.691281 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" podUID="7adb920d-0538-46bb-8c96-3d5f68902e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 17:10:38.643748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:38.642105 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-9s7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:10:38.643748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:38.642170 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" podUID="cd5c0df9-6522-46f6-8222-117b7ad17bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 17:10:38.689209 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:38.689177 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-g2hzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:10:38.689435 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:38.689232 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" podUID="7adb920d-0538-46bb-8c96-3d5f68902e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 17:10:39.643191 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:39.643154 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-9s7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:10:39.643843 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:39.643810 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" podUID="cd5c0df9-6522-46f6-8222-117b7ad17bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 17:10:39.691748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:39.691093 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-g2hzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:10:39.691748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:39.691158 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" podUID="7adb920d-0538-46bb-8c96-3d5f68902e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 17:10:40.639372 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:40.639338 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-9s7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:10:40.639856 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:40.639824 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" podUID="cd5c0df9-6522-46f6-8222-117b7ad17bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 17:10:40.689065 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:40.689035 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-g2hzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:10:40.689428 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:40.689088 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" podUID="7adb920d-0538-46bb-8c96-3d5f68902e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 17:10:41.640136 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:41.640102 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-9s7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 17:10:41.640422 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:41.640391 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" podUID="cd5c0df9-6522-46f6-8222-117b7ad17bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 17:10:41.688774 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:41.688740 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-g2hzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Oct 30 17:10:41.688973 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:41.688800 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" podUID="7adb920d-0538-46bb-8c96-3d5f68902e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Oct 30 17:10:42.816751 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:42.816298 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw"] Oct 30 17:10:42.818316 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:42.817429 4861 cpu_manager.go:414] "containerMap: removing container" podUID="746b39b4-7b4b-4fe3-b9eb-944af7954669" containerName="payload-processing" Oct 30 17:10:42.818316 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:42.817453 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746b39b4-7b4b-4fe3-b9eb-944af7954669" containerName="payload-processing" Oct 30 17:10:42.818316 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:42.817677 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="746b39b4-7b4b-4fe3-b9eb-944af7954669" containerName="payload-processing" Oct 30 17:10:42.825121 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:42.825096 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:10:42.831413 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:42.831372 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw"] Oct 30 17:10:42.870109 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:42.869864 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"payload-processing-external-model-plugins-e2e-extmod-a-f94fb5\"" Oct 30 17:10:42.870109 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:42.869890 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"routing-overlay\"" Oct 30 17:10:42.870109 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:42.869936 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"openshift-service-ca.crt\"" Oct 30 17:10:42.870109 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:42.869973 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"payload-processing-external-model-e2e--ac654823-dockercfg-cbc85\"" Oct 30 17:10:42.870109 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:42.870000 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"agc-cred-90e9f0cba9dc\"" Oct 30 17:10:42.870109 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:42.870017 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"kube-root-ca.crt\"" Oct 30 17:10:42.890581 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:42.890551 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fj66\" (UniqueName: \"kubernetes.io/projected/06ac0898-f2b4-4605-9f7b-2a1dd4628995-kube-api-access-9fj66\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw\" (UID: \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:10:42.891444 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:42.891420 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06ac0898-f2b4-4605-9f7b-2a1dd4628995-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw\" (UID: \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:10:42.891630 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:42.891610 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ac0898-f2b4-4605-9f7b-2a1dd4628995-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw\" (UID: \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:10:42.891801 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:42.891781 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06ac0898-f2b4-4605-9f7b-2a1dd4628995-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw\" (UID: \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:10:43.010746 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.007291 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fj66\" (UniqueName: \"kubernetes.io/projected/06ac0898-f2b4-4605-9f7b-2a1dd4628995-kube-api-access-9fj66\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw\" (UID: \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:10:43.021554 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.021454 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06ac0898-f2b4-4605-9f7b-2a1dd4628995-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw\" (UID: \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:10:43.031786 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.031648 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ac0898-f2b4-4605-9f7b-2a1dd4628995-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw\" (UID: \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:10:43.031786 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.031709 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06ac0898-f2b4-4605-9f7b-2a1dd4628995-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw\" (UID: \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:10:43.032127 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.022296 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06ac0898-f2b4-4605-9f7b-2a1dd4628995-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw\" (UID: \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:10:43.033372 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.033341 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ac0898-f2b4-4605-9f7b-2a1dd4628995-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw\" (UID: \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:10:43.057481 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.057435 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06ac0898-f2b4-4605-9f7b-2a1dd4628995-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw\" (UID: \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:10:43.069789 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.068338 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fj66\" (UniqueName: \"kubernetes.io/projected/06ac0898-f2b4-4605-9f7b-2a1dd4628995-kube-api-access-9fj66\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw\" (UID: \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:10:43.168316 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.168284 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:10:43.532867 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.532827 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw"] Oct 30 17:10:43.542066 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:10:43.542039 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ac0898_f2b4_4605_9f7b_2a1dd4628995.slice/crio-0caaf8327f76b54b958a0ed80d54c24708db1eeb5d318fff88cb630ede743a53 WatchSource:0}: Error finding container 0caaf8327f76b54b958a0ed80d54c24708db1eeb5d318fff88cb630ede743a53: Status 404 returned error can't find the container with id 0caaf8327f76b54b958a0ed80d54c24708db1eeb5d318fff88cb630ede743a53 Oct 30 17:10:43.691252 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.691196 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-9s7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:43.691385 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.691273 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" podUID="cd5c0df9-6522-46f6-8222-117b7ad17bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:43.738877 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.738844 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-g2hzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:43.739090 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.738898 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" podUID="7adb920d-0538-46bb-8c96-3d5f68902e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:43.794679 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.794604 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4"] Oct 30 17:10:43.800954 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.800928 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:10:43.823028 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.822276 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"agc-cred-7adf77356401\"" Oct 30 17:10:43.823028 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.822523 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"payload-processing-external-model-e2e--be0deb1b-dockercfg-k86x2\"" Oct 30 17:10:43.825608 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.825569 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4"] Oct 30 17:10:43.848782 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.828986 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"payload-processing-external-model-plugins-e2e-extmod-b-13c495\"" Oct 30 17:10:43.848782 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.829610 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"openshift-service-ca.crt\"" Oct 30 17:10:43.848782 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.829696 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"kube-root-ca.crt\"" Oct 30 17:10:43.848782 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.829798 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-13c495\"/\"routing-overlay\"" Oct 30 17:10:43.960859 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.959324 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2ec4f78-379e-4f7b-9b08-686be0d0524a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4\" (UID: \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:10:43.960859 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.959393 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2ec4f78-379e-4f7b-9b08-686be0d0524a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4\" (UID: \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:10:43.960859 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.959538 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kglkn\" (UniqueName: \"kubernetes.io/projected/c2ec4f78-379e-4f7b-9b08-686be0d0524a-kube-api-access-kglkn\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4\" (UID: \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:10:43.960859 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:43.959699 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ec4f78-379e-4f7b-9b08-686be0d0524a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4\" (UID: \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:10:44.062738 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.062631 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kglkn\" (UniqueName: \"kubernetes.io/projected/c2ec4f78-379e-4f7b-9b08-686be0d0524a-kube-api-access-kglkn\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4\" (UID: \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:10:44.082173 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.082133 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ec4f78-379e-4f7b-9b08-686be0d0524a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4\" (UID: \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:10:44.083754 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.083702 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2ec4f78-379e-4f7b-9b08-686be0d0524a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4\" (UID: \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:10:44.093529 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.093499 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2ec4f78-379e-4f7b-9b08-686be0d0524a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4\" (UID: \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:10:44.094373 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.084528 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2ec4f78-379e-4f7b-9b08-686be0d0524a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4\" (UID: \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:10:44.100360 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.083434 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ec4f78-379e-4f7b-9b08-686be0d0524a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4\" (UID: \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:10:44.137913 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.132139 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2ec4f78-379e-4f7b-9b08-686be0d0524a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4\" (UID: \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:10:44.137913 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.134400 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kglkn\" (UniqueName: \"kubernetes.io/projected/c2ec4f78-379e-4f7b-9b08-686be0d0524a-kube-api-access-kglkn\") pod \"payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4\" (UID: \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\") " pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:10:44.143408 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.143381 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:10:44.467554 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.467527 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4"] Oct 30 17:10:44.472059 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:10:44.472028 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ec4f78_379e_4f7b_9b08_686be0d0524a.slice/crio-5a507575831cef08bffdf6e74d322f32d4ec758ab8b65ad04fa7b414b124860d WatchSource:0}: Error finding container 5a507575831cef08bffdf6e74d322f32d4ec758ab8b65ad04fa7b414b124860d: Status 404 returned error can't find the container with id 5a507575831cef08bffdf6e74d322f32d4ec758ab8b65ad04fa7b414b124860d Oct 30 17:10:44.556377 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.556340 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" event={"ID":"c2ec4f78-379e-4f7b-9b08-686be0d0524a","Type":"ContainerStarted","Data":"5a507575831cef08bffdf6e74d322f32d4ec758ab8b65ad04fa7b414b124860d"} Oct 30 17:10:44.562335 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.562301 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" event={"ID":"06ac0898-f2b4-4605-9f7b-2a1dd4628995","Type":"ContainerStarted","Data":"4e25a247ff70f49bad9eee07460f0b3a35b65bb1a5d3418b1f5bf6a8352ab816"} Oct 30 17:10:44.562493 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.562360 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" event={"ID":"06ac0898-f2b4-4605-9f7b-2a1dd4628995","Type":"ContainerStarted","Data":"0caaf8327f76b54b958a0ed80d54c24708db1eeb5d318fff88cb630ede743a53"} Oct 30 17:10:44.562493 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.562480 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:10:44.616412 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.616344 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" podStartSLOduration=2.616325749 podStartE2EDuration="2.616325749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:44.614609236 +0000 UTC m=+2874.957283289" watchObservedRunningTime="2026-10-30 17:10:44.616325749 +0000 UTC m=+2874.958999787" Oct 30 17:10:44.691961 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.691916 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-9s7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:44.692150 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.692003 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" podUID="cd5c0df9-6522-46f6-8222-117b7ad17bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:44.759787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.759621 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-g2hzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:44.759787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:44.759682 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" podUID="7adb920d-0538-46bb-8c96-3d5f68902e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:45.594046 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:45.594010 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" event={"ID":"c2ec4f78-379e-4f7b-9b08-686be0d0524a","Type":"ContainerStarted","Data":"85ba0434e882099789468ca6e8bcf12e64c9e00fd98fb195decde1b9d3eedcee"} Oct 30 17:10:45.641747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:45.641422 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" podStartSLOduration=2.641403142 podStartE2EDuration="2.641403142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:45.639568576 +0000 UTC m=+2875.982242619" watchObservedRunningTime="2026-10-30 17:10:45.641403142 +0000 UTC m=+2875.984077182" Oct 30 17:10:45.736769 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:45.733843 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-9s7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:45.736769 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:45.733903 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" podUID="cd5c0df9-6522-46f6-8222-117b7ad17bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:45.806765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:45.803894 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-g2hzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:45.806765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:45.803951 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" podUID="7adb920d-0538-46bb-8c96-3d5f68902e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:46.602208 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:46.602175 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:10:46.776900 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:46.776847 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-9s7vv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:46.777084 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:46.776914 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" podUID="cd5c0df9-6522-46f6-8222-117b7ad17bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:46.845922 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:46.845885 4861 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7854949c9-g2hzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:10:46.846156 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:46.846133 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" podUID="7adb920d-0538-46bb-8c96-3d5f68902e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:10:46.880186 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:46.880104 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:46.880876 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:46.880850 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:46.881678 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:46.881652 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:10:46.943662 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:46.943621 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:46.944122 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:46.944098 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:46.945329 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:46.945308 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:10:50.298086 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:10:50.298048 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:10:51.523914 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.523499 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc"] Oct 30 17:10:51.790015 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.789852 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:10:51.836567 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.836529 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"openshift-service-ca.crt\"" Oct 30 17:10:51.837371 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.837347 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"payload-processing-external-model-e2e--5c3004c8-dockercfg-nch74\"" Oct 30 17:10:51.837705 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.837685 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"agc-cred-76749d78aca4\"" Oct 30 17:10:51.895754 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.895704 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc\" (UID: \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:10:51.896006 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.895982 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc\" (UID: \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:10:51.896161 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.896144 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc\" (UID: \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:10:51.896434 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.896416 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxh42\" (UniqueName: \"kubernetes.io/projected/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-kube-api-access-bxh42\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc\" (UID: \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:10:51.897201 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.897178 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"kube-root-ca.crt\"" Oct 30 17:10:51.897842 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.897800 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5758b7\"" Oct 30 17:10:51.910035 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.899227 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"routing-overlay\"" Oct 30 17:10:51.910035 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.899286 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5758b7\"/\"agc-cred-0d17396f5b80\"" Oct 30 17:10:51.997761 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.997647 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc\" (UID: \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:10:51.998056 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.997765 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc\" (UID: \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:10:51.998056 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.997954 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxh42\" (UniqueName: \"kubernetes.io/projected/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-kube-api-access-bxh42\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc\" (UID: \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:10:51.998209 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.998145 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc\" (UID: \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:10:51.999830 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.999411 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc\" (UID: \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:10:51.999830 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:51.999411 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc\" (UID: \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:10:52.044780 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:52.034802 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxh42\" (UniqueName: \"kubernetes.io/projected/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-kube-api-access-bxh42\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc\" (UID: \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:10:52.052751 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:52.051450 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc\" (UID: \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\") " pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:10:52.086651 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:52.086624 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc"] Oct 30 17:10:52.088525 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:52.088506 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:10:52.670607 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:52.670579 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc"] Oct 30 17:10:53.088807 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:53.088776 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" event={"ID":"f4c05e79-f99c-4b1b-8a48-3e366b6f5472","Type":"ContainerStarted","Data":"18d264f58dbf569d3571194d4bce0027e584894b63e992c3b72be94bfb7c91be"} Oct 30 17:10:53.088981 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:53.088828 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" event={"ID":"f4c05e79-f99c-4b1b-8a48-3e366b6f5472","Type":"ContainerStarted","Data":"fdd0edaa11fafad0e20e46b3af8e431a022e6666385a488e8cc1090d5df8fe5c"} Oct 30 17:10:53.088981 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:53.088857 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:10:53.136468 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:53.136421 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" podStartSLOduration=2.136408591 podStartE2EDuration="2.136408591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:53.136143317 +0000 UTC m=+2883.478817357" watchObservedRunningTime="2026-10-30 17:10:53.136408591 +0000 UTC m=+2883.479082675" Oct 30 17:10:55.598608 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:55.598577 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:10:57.624259 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:57.624227 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:10:58.996701 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:58.996661 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk"] Oct 30 17:10:59.002246 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.002215 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:10:59.021148 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.021120 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"kube-root-ca.crt\"" Oct 30 17:10:59.021148 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.021120 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-43197b\"" Oct 30 17:10:59.037227 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.035286 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"openshift-service-ca.crt\"" Oct 30 17:10:59.037227 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.035444 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"agc-cred-fa509db993dc\"" Oct 30 17:10:59.037227 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.035530 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"payload-processing-external-model-e2e--e2ae9620-dockercfg-kmxss\"" Oct 30 17:10:59.037227 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.035654 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-43197b\"/\"routing-overlay\"" Oct 30 17:10:59.042529 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.042503 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk"] Oct 30 17:10:59.086378 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.083910 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tmzx\" (UniqueName: \"kubernetes.io/projected/c263ff4a-fa28-4eaa-9d93-75a2923f9161-kube-api-access-2tmzx\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk\" (UID: \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:10:59.086378 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.084117 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c263ff4a-fa28-4eaa-9d93-75a2923f9161-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk\" (UID: \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:10:59.086378 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.084177 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c263ff4a-fa28-4eaa-9d93-75a2923f9161-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk\" (UID: \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:10:59.086378 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.084219 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c263ff4a-fa28-4eaa-9d93-75a2923f9161-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk\" (UID: \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:10:59.185786 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.185716 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tmzx\" (UniqueName: \"kubernetes.io/projected/c263ff4a-fa28-4eaa-9d93-75a2923f9161-kube-api-access-2tmzx\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk\" (UID: \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:10:59.186306 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.186275 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c263ff4a-fa28-4eaa-9d93-75a2923f9161-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk\" (UID: \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:10:59.186436 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.186352 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c263ff4a-fa28-4eaa-9d93-75a2923f9161-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk\" (UID: \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:10:59.186436 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.186378 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c263ff4a-fa28-4eaa-9d93-75a2923f9161-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk\" (UID: \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:10:59.189068 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.189040 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c263ff4a-fa28-4eaa-9d93-75a2923f9161-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk\" (UID: \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:10:59.190572 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.190547 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c263ff4a-fa28-4eaa-9d93-75a2923f9161-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk\" (UID: \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:10:59.200082 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.200054 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c263ff4a-fa28-4eaa-9d93-75a2923f9161-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk\" (UID: \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:10:59.217354 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.217300 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tmzx\" (UniqueName: \"kubernetes.io/projected/c263ff4a-fa28-4eaa-9d93-75a2923f9161-kube-api-access-2tmzx\") pod \"payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk\" (UID: \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\") " pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:10:59.323250 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.323126 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:10:59.575217 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.575140 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl"] Oct 30 17:10:59.581359 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.581334 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.604694 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.604666 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl"] Oct 30 17:10:59.613382 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.613344 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-3ca339-openshift-default-dockercfg-nv7n7\"" Oct 30 17:10:59.646019 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:10:59.645960 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc263ff4a_fa28_4eaa_9d93_75a2923f9161.slice/crio-52476be5a7a12b6eeefdf599583805526fcec30acaaf3dbb8c61168ff8555c56 WatchSource:0}: Error finding container 52476be5a7a12b6eeefdf599583805526fcec30acaaf3dbb8c61168ff8555c56: Status 404 returned error can't find the container with id 52476be5a7a12b6eeefdf599583805526fcec30acaaf3dbb8c61168ff8555c56 Oct 30 17:10:59.646251 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.646152 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk"] Oct 30 17:10:59.695188 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.695160 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-token\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.695301 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.695210 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-data\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.695363 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.695308 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-credential-socket\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.695480 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.695453 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-workload-socket\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.695601 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.695581 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-envoy\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.695652 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.695615 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b7h8\" (UniqueName: \"kubernetes.io/projected/9e67b1b9-0721-4914-85ed-6abf64720a33-kube-api-access-6b7h8\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.695652 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.695641 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e67b1b9-0721-4914-85ed-6abf64720a33-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.695833 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.695815 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-workload-certs\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.695898 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.695867 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-podinfo\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.828802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.803709 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e67b1b9-0721-4914-85ed-6abf64720a33-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.828802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.803880 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-workload-certs\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.828802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.803938 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-podinfo\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.828802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.804068 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-token\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.828802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.804112 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-data\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.828802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.804587 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-data\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.828802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.804662 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-credential-socket\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.828802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.804788 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-workload-socket\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.828802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.804910 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-envoy\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.828802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.804953 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7h8\" (UniqueName: \"kubernetes.io/projected/9e67b1b9-0721-4914-85ed-6abf64720a33-kube-api-access-6b7h8\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.828802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.805832 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-credential-socket\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.828802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.806099 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-workload-socket\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.828802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.813540 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e67b1b9-0721-4914-85ed-6abf64720a33-istiod-ca-cert\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.828802 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.813813 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-workload-certs\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.871716 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.858377 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-token\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.876071 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.876037 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-podinfo\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.876589 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.876561 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-envoy\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.880156 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.880069 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b7h8\" (UniqueName: \"kubernetes.io/projected/9e67b1b9-0721-4914-85ed-6abf64720a33-kube-api-access-6b7h8\") pod \"e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:10:59.899993 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:10:59.899961 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:11:00.209392 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:00.209289 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" event={"ID":"c263ff4a-fa28-4eaa-9d93-75a2923f9161","Type":"ContainerStarted","Data":"4988e924f8095dbcd8b9e794ae2f2636cbfe08d3e4c1da9e44e03344aa0f36c2"} Oct 30 17:11:00.209392 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:00.209360 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" event={"ID":"c263ff4a-fa28-4eaa-9d93-75a2923f9161","Type":"ContainerStarted","Data":"52476be5a7a12b6eeefdf599583805526fcec30acaaf3dbb8c61168ff8555c56"} Oct 30 17:11:00.209865 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:00.209514 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:11:00.235240 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:00.235210 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl"] Oct 30 17:11:00.235607 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:11:00.235581 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e67b1b9_0721_4914_85ed_6abf64720a33.slice/crio-1a3a678ce527f9d754feae713a0128d23a8f29cbee43247b0159c6b44f5b3ac1 WatchSource:0}: Error finding container 1a3a678ce527f9d754feae713a0128d23a8f29cbee43247b0159c6b44f5b3ac1: Status 404 returned error can't find the container with id 1a3a678ce527f9d754feae713a0128d23a8f29cbee43247b0159c6b44f5b3ac1 Oct 30 17:11:00.238859 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:00.238826 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:00.238937 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:00.238897 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:00.238996 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:00.238933 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:00.262100 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:00.262037 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" podStartSLOduration=2.262020612 podStartE2EDuration="2.262020612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:00.259988446 +0000 UTC m=+2890.602662487" watchObservedRunningTime="2026-10-30 17:11:00.262020612 +0000 UTC m=+2890.604694653" Oct 30 17:11:01.219772 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:01.219708 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" event={"ID":"9e67b1b9-0721-4914-85ed-6abf64720a33","Type":"ContainerStarted","Data":"70137b579187c78386389b70f9cebdd26ec7de8cd220f414531a79332306d5b3"} Oct 30 17:11:01.220194 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:01.219791 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" event={"ID":"9e67b1b9-0721-4914-85ed-6abf64720a33","Type":"ContainerStarted","Data":"1a3a678ce527f9d754feae713a0128d23a8f29cbee43247b0159c6b44f5b3ac1"} Oct 30 17:11:01.900142 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:01.900108 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:11:01.927131 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:01.927089 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:11:01.927424 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:01.927395 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" podUID="9e67b1b9-0721-4914-85ed-6abf64720a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 17:11:02.904757 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:02.901055 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:11:02.904757 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:02.901119 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" podUID="9e67b1b9-0721-4914-85ed-6abf64720a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 17:11:02.932167 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:02.932098 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" podStartSLOduration=3.9320764820000003 podStartE2EDuration="3.932076482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:01.2720764 +0000 UTC m=+2891.614750439" watchObservedRunningTime="2026-10-30 17:11:02.932076482 +0000 UTC m=+2893.274750521" Oct 30 17:11:02.936221 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:02.936195 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l"] Oct 30 17:11:02.941412 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:02.941393 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:02.959521 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:02.959500 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-13c495-openshift-default-dockercfg-pnbrr\"" Oct 30 17:11:02.980041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:02.980009 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l"] Oct 30 17:11:02.986403 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:02.986375 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-istio-envoy\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:02.986551 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:02.986429 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20bdae97-71b8-4376-85b4-157138986bb2-istiod-ca-cert\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:02.986551 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:02.986513 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-workload-socket\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:02.986679 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:02.986625 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-workload-certs\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:02.986774 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:02.986754 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg4w4\" (UniqueName: \"kubernetes.io/projected/20bdae97-71b8-4376-85b4-157138986bb2-kube-api-access-fg4w4\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:02.986854 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:02.986835 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-credential-socket\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:02.986903 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:02.986870 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20bdae97-71b8-4376-85b4-157138986bb2-istio-token\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:02.986958 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:02.986937 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20bdae97-71b8-4376-85b4-157138986bb2-istio-podinfo\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:02.987081 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:02.987063 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-istio-data\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.088483 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.088448 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-istio-envoy\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.088483 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.088487 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20bdae97-71b8-4376-85b4-157138986bb2-istiod-ca-cert\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.088760 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.088503 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-workload-socket\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.088760 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.088552 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-workload-certs\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.088760 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.088602 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg4w4\" (UniqueName: \"kubernetes.io/projected/20bdae97-71b8-4376-85b4-157138986bb2-kube-api-access-fg4w4\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.088760 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.088652 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-credential-socket\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.088760 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.088669 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20bdae97-71b8-4376-85b4-157138986bb2-istio-token\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.088760 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.088687 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20bdae97-71b8-4376-85b4-157138986bb2-istio-podinfo\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.089100 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.088775 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-istio-data\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.089100 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.088997 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-workload-certs\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.089100 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.089059 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-workload-socket\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.089269 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.089246 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-istio-data\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.089581 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.089556 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20bdae97-71b8-4376-85b4-157138986bb2-istiod-ca-cert\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.089652 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.089564 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-credential-socket\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.091153 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.091130 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-istio-envoy\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.091625 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.091601 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20bdae97-71b8-4376-85b4-157138986bb2-istio-podinfo\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.131745 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.129706 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20bdae97-71b8-4376-85b4-157138986bb2-istio-token\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.132279 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.132246 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg4w4\" (UniqueName: \"kubernetes.io/projected/20bdae97-71b8-4376-85b4-157138986bb2-kube-api-access-fg4w4\") pod \"e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.257753 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.257520 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:03.532264 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.532181 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v"] Oct 30 17:11:03.538525 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.538504 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.560055 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.560029 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v"] Oct 30 17:11:03.568565 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.568540 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a77b48-openshift-default-dockercfg-725dg\"" Oct 30 17:11:03.598423 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.598382 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.598632 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.598478 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.598632 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.598533 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.598632 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.598551 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q44k\" (UniqueName: \"kubernetes.io/projected/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-kube-api-access-2q44k\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.598632 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.598570 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.598632 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.598611 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.599021 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.598784 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.599021 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.598868 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.599021 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.598984 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.647077 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.647052 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l"] Oct 30 17:11:03.648465 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:11:03.648440 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20bdae97_71b8_4376_85b4_157138986bb2.slice/crio-e29b48292a530e100fbca37db02a74c00d5d87b11df3bcad7f5477b7fec23a08 WatchSource:0}: Error finding container e29b48292a530e100fbca37db02a74c00d5d87b11df3bcad7f5477b7fec23a08: Status 404 returned error can't find the container with id e29b48292a530e100fbca37db02a74c00d5d87b11df3bcad7f5477b7fec23a08 Oct 30 17:11:03.652099 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.652063 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:03.652199 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.652142 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:03.652240 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.652208 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:03.700106 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.699893 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.700106 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.699940 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.700106 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.699972 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.700106 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.700077 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.700436 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.700156 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.700436 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.700215 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.700436 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.700245 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q44k\" (UniqueName: \"kubernetes.io/projected/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-kube-api-access-2q44k\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.700436 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.700277 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.700436 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.700317 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.700436 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.700335 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-data\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.700835 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.700494 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-credential-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.705830 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.701016 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istiod-ca-cert\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.705830 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.701492 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-workload-certs\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.705830 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.701750 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-workload-socket\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.722577 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.722551 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-podinfo\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.734075 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.733990 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-envoy\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.735244 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.734566 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-token\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.737543 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.737517 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q44k\" (UniqueName: \"kubernetes.io/projected/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-kube-api-access-2q44k\") pod \"e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.854335 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.854254 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:03.905594 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.905136 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:11:03.905594 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:03.905203 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" podUID="9e67b1b9-0721-4914-85ed-6abf64720a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 17:11:04.145095 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.145018 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:11:04.269912 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.269370 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v"] Oct 30 17:11:04.273636 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.273606 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:04.273842 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.273815 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:04.273989 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.273974 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:04.354361 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.354317 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" event={"ID":"20bdae97-71b8-4376-85b4-157138986bb2","Type":"ContainerStarted","Data":"14317835bc3ae9b0e2fd12eb31bd18903ecc3fa4ace244d35069fd281cdf1d99"} Oct 30 17:11:04.354505 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.354459 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" event={"ID":"20bdae97-71b8-4376-85b4-157138986bb2","Type":"ContainerStarted","Data":"e29b48292a530e100fbca37db02a74c00d5d87b11df3bcad7f5477b7fec23a08"} Oct 30 17:11:04.358143 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.358107 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" event={"ID":"26f54ab0-924f-4d9d-aa69-69e9015b1bd5","Type":"ContainerStarted","Data":"f1dc08ab59e37b32b5a9a806643ceb425c482920a84970639ff9e56f23dbcb40"} Oct 30 17:11:04.635749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.626774 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" podStartSLOduration=2.626754938 podStartE2EDuration="2.626754938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:04.411676863 +0000 UTC m=+2894.754350902" watchObservedRunningTime="2026-10-30 17:11:04.626754938 +0000 UTC m=+2894.969428977" Oct 30 17:11:04.671879 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.671776 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr"] Oct 30 17:11:04.681209 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.680087 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.681209 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.680989 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr"] Oct 30 17:11:04.732074 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.732041 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-18519c-openshift-default-dockercfg-nqftk\"" Oct 30 17:11:04.876746 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.876685 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.876971 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.876822 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvz7c\" (UniqueName: \"kubernetes.io/projected/854e3a2c-0257-4f16-96ba-e86cc2ba282d-kube-api-access-lvz7c\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.876971 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.876856 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.877112 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.877100 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.877178 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.877144 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.877240 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.877192 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.877300 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.877253 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.877300 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.877289 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.877420 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.877326 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.929856 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.905062 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:11:04.930460 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.930416 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" podUID="9e67b1b9-0721-4914-85ed-6abf64720a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 17:11:04.978836 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.978799 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.979025 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.978866 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.979025 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.978922 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.979025 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.978986 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.979137 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.979036 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.979137 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.979075 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.979317 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.979294 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.979373 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.979344 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvz7c\" (UniqueName: \"kubernetes.io/projected/854e3a2c-0257-4f16-96ba-e86cc2ba282d-kube-api-access-lvz7c\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.979407 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.979383 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.979847 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.979788 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-workload-certs\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.979847 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.979811 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-credential-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.980075 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.979986 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-workload-socket\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.980365 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.980344 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-data\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.980815 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.980794 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istiod-ca-cert\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.982170 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.982148 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-envoy\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:04.985850 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:04.985825 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-podinfo\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:05.026797 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:05.023376 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-token\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:05.026797 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:05.026434 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvz7c\" (UniqueName: \"kubernetes.io/projected/854e3a2c-0257-4f16-96ba-e86cc2ba282d-kube-api-access-lvz7c\") pod \"e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:05.046761 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:05.046015 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:05.266789 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:05.266139 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:05.274121 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:05.274085 4861 patch_prober.go:29] interesting pod/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 17:11:05.274317 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:05.274148 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" podUID="20bdae97-71b8-4376-85b4-157138986bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 17:11:05.378384 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:05.378343 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" event={"ID":"26f54ab0-924f-4d9d-aa69-69e9015b1bd5","Type":"ContainerStarted","Data":"0c07378670c3f0e4b55498a44b69ac5885300af96c2d215acd4b24ee89b66e86"} Oct 30 17:11:05.555273 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:05.554417 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr"] Oct 30 17:11:05.784987 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:05.779338 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:05.784987 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:05.779428 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:05.784987 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:05.779471 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:05.874860 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:05.869763 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 17:11:05.874860 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:05.869820 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" podUID="26f54ab0-924f-4d9d-aa69-69e9015b1bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 17:11:05.901528 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:05.900527 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:11:05.901528 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:05.900686 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" podUID="9e67b1b9-0721-4914-85ed-6abf64720a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 17:11:06.082983 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.082916 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" podStartSLOduration=3.082894688 podStartE2EDuration="3.082894688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:05.55139036 +0000 UTC m=+2895.894064393" watchObservedRunningTime="2026-10-30 17:11:06.082894688 +0000 UTC m=+2896.425568729" Oct 30 17:11:06.192331 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.192296 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:06.192683 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.192666 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth"] Oct 30 17:11:06.200136 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.199527 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth"] Oct 30 17:11:06.200136 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.199646 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.266639 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.265146 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-f94fb5-openshift-default-dockercfg-9vtqh\"" Oct 30 17:11:06.288084 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.287878 4861 patch_prober.go:29] interesting pod/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 17:11:06.288084 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.287938 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" podUID="20bdae97-71b8-4376-85b4-157138986bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 17:11:06.291846 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.291816 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cfda264-9b06-4846-8245-394666f98198-istio-token\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.291998 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.291893 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cfda264-9b06-4846-8245-394666f98198-istiod-ca-cert\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.291998 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.291938 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-workload-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.291998 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.291983 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cfda264-9b06-4846-8245-394666f98198-istio-podinfo\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.292132 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.292057 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-istio-envoy\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.292173 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.292153 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-istio-data\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.292228 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.292210 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-credential-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.292287 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.292272 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-workload-certs\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.292379 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.292361 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2hdk\" (UniqueName: \"kubernetes.io/projected/9cfda264-9b06-4846-8245-394666f98198-kube-api-access-s2hdk\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.398753 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.398593 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-istio-envoy\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.398753 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.398742 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-istio-data\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.399027 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.398803 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-credential-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.399027 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.398859 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-workload-certs\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.399027 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.398942 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2hdk\" (UniqueName: \"kubernetes.io/projected/9cfda264-9b06-4846-8245-394666f98198-kube-api-access-s2hdk\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.399212 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.399071 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cfda264-9b06-4846-8245-394666f98198-istio-token\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.399212 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.399127 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cfda264-9b06-4846-8245-394666f98198-istiod-ca-cert\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.399212 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.399170 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-workload-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.399400 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.399213 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cfda264-9b06-4846-8245-394666f98198-istio-podinfo\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.400234 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.400200 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-workload-certs\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.418815 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.401578 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cfda264-9b06-4846-8245-394666f98198-istiod-ca-cert\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.422756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.419114 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-istio-data\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.422756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.419441 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-credential-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.444813 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.423791 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-workload-socket\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.444813 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.426623 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" event={"ID":"854e3a2c-0257-4f16-96ba-e86cc2ba282d","Type":"ContainerStarted","Data":"20c14f342e77425f62bf27285449aa758b877b0df8e4a4934aa08108bdf32a2b"} Oct 30 17:11:06.488442 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.488326 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cfda264-9b06-4846-8245-394666f98198-istio-podinfo\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.490557 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.489601 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-istio-envoy\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.493770 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.492436 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2hdk\" (UniqueName: \"kubernetes.io/projected/9cfda264-9b06-4846-8245-394666f98198-kube-api-access-s2hdk\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.494474 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.494419 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cfda264-9b06-4846-8245-394666f98198-istio-token\") pod \"e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.682801 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.682625 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:06.914082 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.914045 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 17:11:06.914362 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.914335 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" podUID="26f54ab0-924f-4d9d-aa69-69e9015b1bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 17:11:06.916100 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.916072 4861 patch_prober.go:29] interesting pod/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" start-of-body= Oct 30 17:11:06.916299 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:06.916277 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" podUID="9e67b1b9-0721-4914-85ed-6abf64720a33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.10:15021/healthz/ready\": dial tcp 10.132.1.10:15021: connect: connection refused" Oct 30 17:11:07.281295 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:07.281190 4861 patch_prober.go:29] interesting pod/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 17:11:07.282380 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:07.281805 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" podUID="20bdae97-71b8-4376-85b4-157138986bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 17:11:07.390577 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:07.390531 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth"] Oct 30 17:11:07.479941 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:07.478552 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:07.479941 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:07.478663 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:07.479941 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:07.478707 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:07.490548 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:07.490513 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" event={"ID":"854e3a2c-0257-4f16-96ba-e86cc2ba282d","Type":"ContainerStarted","Data":"9ad0a003cd360964e6b65e6b9da3d895a4ee1cf3279657002acd9c0400694271"} Oct 30 17:11:07.495113 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:07.495082 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" event={"ID":"9cfda264-9b06-4846-8245-394666f98198","Type":"ContainerStarted","Data":"97c43ebe3f0ae54532938101ad67549e286434f705e13e3557835613c6242638"} Oct 30 17:11:07.651771 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:07.650369 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" podStartSLOduration=3.650346876 podStartE2EDuration="3.650346876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:07.620042105 +0000 UTC m=+2897.962716145" watchObservedRunningTime="2026-10-30 17:11:07.650346876 +0000 UTC m=+2897.993020919" Oct 30 17:11:07.865823 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:07.865064 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 17:11:07.865823 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:07.865126 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" podUID="26f54ab0-924f-4d9d-aa69-69e9015b1bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 17:11:08.053749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:08.052858 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 17:11:08.053749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:08.052930 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" podUID="854e3a2c-0257-4f16-96ba-e86cc2ba282d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 17:11:08.194834 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:08.192347 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:11:08.194834 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:08.192423 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:08.427359 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:08.427320 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:11:08.453707 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:08.436533 4861 patch_prober.go:29] interesting pod/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 17:11:08.453707 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:08.436593 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" podUID="20bdae97-71b8-4376-85b4-157138986bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 17:11:08.722694 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:08.722666 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:11:09.025812 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:08.849313 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:09.025812 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:08.987843 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:09.027877 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.027837 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b"] Oct 30 17:11:09.068633 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.048656 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 17:11:09.068633 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.048740 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" podUID="26f54ab0-924f-4d9d-aa69-69e9015b1bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 17:11:09.073931 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.069058 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 17:11:09.073931 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.069124 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" podUID="854e3a2c-0257-4f16-96ba-e86cc2ba282d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 17:11:09.094756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.091163 4861 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:09.181399 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.181364 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.248753 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.248310 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-420ff3-openshift-default-dockercfg-mlfh4\"" Oct 30 17:11:09.282064 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.281964 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.321617 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.321584 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.321908 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.321866 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.322143 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.322120 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.339931 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.339894 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.340337 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.340314 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.379011 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.378964 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.379415 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.379389 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.379620 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.379604 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdmkr\" (UniqueName: \"kubernetes.io/projected/24e3b70b-25df-4464-be5e-d5f77e1db8fd-kube-api-access-tdmkr\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.442650 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.442609 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b"] Oct 30 17:11:09.522748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.502814 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 17:11:09.522748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.504184 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.522748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.504257 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.522748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.504309 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdmkr\" (UniqueName: \"kubernetes.io/projected/24e3b70b-25df-4464-be5e-d5f77e1db8fd-kube-api-access-tdmkr\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.522748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.504376 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.522748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.504404 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.522748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.504438 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.522748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.504516 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.522748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.504604 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.522748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.504783 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.522748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.505414 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-workload-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.522748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.506487 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istiod-ca-cert\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.597524 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.529873 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" event={"ID":"9cfda264-9b06-4846-8245-394666f98198","Type":"ContainerStarted","Data":"82d8f0afc7cbdd937cf275f538f534f3002d34bceb31d2f351a26ea4522bd175"} Oct 30 17:11:09.597524 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.589014 4861 patch_prober.go:29] interesting pod/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 17:11:09.597524 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.589076 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" podUID="20bdae97-71b8-4376-85b4-157138986bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 17:11:09.598187 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.598126 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-credential-socket\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.619750 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.614926 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-workload-certs\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.654414 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.623859 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-data\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.654564 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.640199 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdmkr\" (UniqueName: \"kubernetes.io/projected/24e3b70b-25df-4464-be5e-d5f77e1db8fd-kube-api-access-tdmkr\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.681239 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.681164 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" podStartSLOduration=4.681128798 podStartE2EDuration="4.681128798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:09.640699565 +0000 UTC m=+2899.983373604" watchObservedRunningTime="2026-10-30 17:11:09.681128798 +0000 UTC m=+2900.023802840" Oct 30 17:11:09.685044 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.685012 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:09.716747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.711296 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-token\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.716747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.715276 4861 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 17:11:09.716747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.715333 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" podUID="9cfda264-9b06-4846-8245-394666f98198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 17:11:09.787751 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.780459 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-podinfo\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.845881 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.845831 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:09.857800 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.857361 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 17:11:09.857800 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.857422 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" podUID="26f54ab0-924f-4d9d-aa69-69e9015b1bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 17:11:09.858057 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.858019 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-envoy\") pod \"e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:09.876928 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:09.876870 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:10.050187 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:10.050144 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 17:11:10.050489 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:10.050456 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" podUID="854e3a2c-0257-4f16-96ba-e86cc2ba282d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 17:11:10.070504 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:10.070469 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:10.286794 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:10.283270 4861 patch_prober.go:29] interesting pod/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Oct 30 17:11:10.286794 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:10.283335 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" podUID="20bdae97-71b8-4376-85b4-157138986bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Oct 30 17:11:10.514879 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:10.514832 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:10.516387 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:10.516361 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 17:11:10.715751 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:10.713538 4861 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 17:11:10.715751 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:10.713598 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" podUID="9cfda264-9b06-4846-8245-394666f98198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 17:11:10.741749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:10.738750 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd" podUID="d37ba69c-1fa5-4f9b-a754-80057e2608b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:10.905805 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:10.889085 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 17:11:10.909843 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:10.909802 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" podUID="26f54ab0-924f-4d9d-aa69-69e9015b1bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 17:11:10.913814 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:10.909462 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 17:11:11.053410 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:11.053374 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 17:11:11.058697 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:11.058661 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" podUID="854e3a2c-0257-4f16-96ba-e86cc2ba282d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 17:11:11.290977 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:11.290947 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:11:11.345697 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:11.345444 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:11.347668 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:11.346445 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:11.386801 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:11.353997 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:11:11.676412 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:11.654290 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:11.676412 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:11.654415 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:11.676412 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:11.654459 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:11:11.768704 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:11.742814 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:11.820780 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:11.814540 4861 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 17:11:11.820780 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:11.814597 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" podUID="9cfda264-9b06-4846-8245-394666f98198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 17:11:11.885839 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:11.871164 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 17:11:11.885839 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:11.883511 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 17:11:11.885839 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:11.883563 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" podUID="26f54ab0-924f-4d9d-aa69-69e9015b1bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 17:11:12.074076 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:12.074031 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 17:11:12.074326 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:12.074295 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" podUID="854e3a2c-0257-4f16-96ba-e86cc2ba282d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 17:11:12.081198 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:12.081166 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b"] Oct 30 17:11:12.099350 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:12.099307 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" event={"ID":"24e3b70b-25df-4464-be5e-d5f77e1db8fd","Type":"ContainerStarted","Data":"21958029dd37115b313f1a6bd659c3be5b03fa191c9d9341a4829cc8e82f6914"} Oct 30 17:11:12.688897 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:12.688441 4861 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 17:11:12.688897 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:12.688521 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" podUID="9cfda264-9b06-4846-8245-394666f98198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 17:11:12.888977 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:12.888418 4861 patch_prober.go:29] interesting pod/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" start-of-body= Oct 30 17:11:12.888977 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:12.888472 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" podUID="26f54ab0-924f-4d9d-aa69-69e9015b1bd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.12:15021/healthz/ready\": dial tcp 10.132.1.12:15021: connect: connection refused" Oct 30 17:11:13.050750 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:13.049871 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 17:11:13.050750 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:13.049934 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" podUID="854e3a2c-0257-4f16-96ba-e86cc2ba282d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 17:11:13.164751 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:13.164190 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" event={"ID":"24e3b70b-25df-4464-be5e-d5f77e1db8fd","Type":"ContainerStarted","Data":"bf39ab140b9a86374f6e5dde7da50e0fe5a64db959fb5e25ff4bd7b56ff95a68"} Oct 30 17:11:13.235756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:13.233816 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" podStartSLOduration=5.233792786 podStartE2EDuration="5.233792786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:13.223242258 +0000 UTC m=+2903.565916298" watchObservedRunningTime="2026-10-30 17:11:13.233792786 +0000 UTC m=+2903.576466825" Oct 30 17:11:13.715750 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:13.713566 4861 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 17:11:13.715750 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:13.713628 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" podUID="9cfda264-9b06-4846-8245-394666f98198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 17:11:14.011769 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:14.010789 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:14.013470 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:14.013445 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:14.028750 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:14.028095 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:11:14.072557 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:14.070359 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 17:11:14.072557 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:14.070415 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" podUID="854e3a2c-0257-4f16-96ba-e86cc2ba282d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 17:11:14.078395 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:14.078369 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:14.086507 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:14.084792 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 17:11:14.086507 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:14.084848 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" podUID="24e3b70b-25df-4464-be5e-d5f77e1db8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 17:11:14.684214 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:14.684185 4861 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 17:11:14.684452 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:14.684431 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" podUID="9cfda264-9b06-4846-8245-394666f98198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 17:11:15.080862 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:15.080824 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 17:11:15.081416 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:15.081387 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" podUID="24e3b70b-25df-4464-be5e-d5f77e1db8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 17:11:15.083745 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:15.080927 4861 patch_prober.go:29] interesting pod/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 30 17:11:15.083949 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:15.083925 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" podUID="854e3a2c-0257-4f16-96ba-e86cc2ba282d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 30 17:11:15.684178 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:15.684143 4861 patch_prober.go:29] interesting pod/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 17:11:15.684420 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:15.684398 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" podUID="9cfda264-9b06-4846-8245-394666f98198" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 17:11:16.074289 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:16.074258 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 17:11:16.074466 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:16.074311 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" podUID="24e3b70b-25df-4464-be5e-d5f77e1db8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 17:11:16.283091 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:16.283060 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:16.283989 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:16.283934 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:16.284966 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:16.284945 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:11:16.689504 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:16.689474 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:16.689913 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:16.689885 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:16.690520 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:16.690504 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:11:17.074198 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:17.074166 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 17:11:17.074382 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:17.074216 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" podUID="24e3b70b-25df-4464-be5e-d5f77e1db8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 17:11:18.074636 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:18.074604 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 17:11:18.075090 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:18.074655 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" podUID="24e3b70b-25df-4464-be5e-d5f77e1db8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 17:11:19.075004 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:19.074959 4861 patch_prober.go:29] interesting pod/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 17:11:19.075479 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:19.075025 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" podUID="24e3b70b-25df-4464-be5e-d5f77e1db8fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 17:11:20.312975 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:11:20.312933 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:11:20.481354 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:20.481322 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:20.481968 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:20.481941 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:20.484448 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:20.484424 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:11:50.293538 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:11:50.293476 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:11:58.489373 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.489337 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn"] Oct 30 17:11:58.495119 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.495094 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:11:58.516228 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.516199 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn"] Oct 30 17:11:58.523982 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.523949 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"routing-overlay\"" Oct 30 17:11:58.524135 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.524078 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5c6b3c\"" Oct 30 17:11:58.524135 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.524127 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"openshift-service-ca.crt\"" Oct 30 17:11:58.524267 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.524174 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"kube-root-ca.crt\"" Oct 30 17:11:58.525218 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.525195 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"agc-cred-a637efc8d26c\"" Oct 30 17:11:58.525342 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.525239 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"payload-processing-external-model-e2e--22b3ecb4-dockercfg-cxhbd\"" Oct 30 17:11:58.671817 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.661816 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv2gl\" (UniqueName: \"kubernetes.io/projected/af3d6e8a-e6de-4423-b316-0b9ef474905e-kube-api-access-bv2gl\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn\" (UID: \"af3d6e8a-e6de-4423-b316-0b9ef474905e\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:11:58.672355 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.672320 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af3d6e8a-e6de-4423-b316-0b9ef474905e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn\" (UID: \"af3d6e8a-e6de-4423-b316-0b9ef474905e\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:11:58.695749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.673041 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af3d6e8a-e6de-4423-b316-0b9ef474905e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn\" (UID: \"af3d6e8a-e6de-4423-b316-0b9ef474905e\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:11:58.695749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.674848 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3d6e8a-e6de-4423-b316-0b9ef474905e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn\" (UID: \"af3d6e8a-e6de-4423-b316-0b9ef474905e\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:11:58.784342 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.777890 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af3d6e8a-e6de-4423-b316-0b9ef474905e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn\" (UID: \"af3d6e8a-e6de-4423-b316-0b9ef474905e\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:11:58.784342 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.777950 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af3d6e8a-e6de-4423-b316-0b9ef474905e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn\" (UID: \"af3d6e8a-e6de-4423-b316-0b9ef474905e\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:11:58.784342 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.779647 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3d6e8a-e6de-4423-b316-0b9ef474905e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn\" (UID: \"af3d6e8a-e6de-4423-b316-0b9ef474905e\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:11:58.784342 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.779946 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2gl\" (UniqueName: \"kubernetes.io/projected/af3d6e8a-e6de-4423-b316-0b9ef474905e-kube-api-access-bv2gl\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn\" (UID: \"af3d6e8a-e6de-4423-b316-0b9ef474905e\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:11:58.784342 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.781108 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af3d6e8a-e6de-4423-b316-0b9ef474905e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn\" (UID: \"af3d6e8a-e6de-4423-b316-0b9ef474905e\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:11:58.784342 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.781180 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3d6e8a-e6de-4423-b316-0b9ef474905e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn\" (UID: \"af3d6e8a-e6de-4423-b316-0b9ef474905e\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:11:58.796761 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.794791 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af3d6e8a-e6de-4423-b316-0b9ef474905e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn\" (UID: \"af3d6e8a-e6de-4423-b316-0b9ef474905e\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:11:58.807033 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.806996 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv2gl\" (UniqueName: \"kubernetes.io/projected/af3d6e8a-e6de-4423-b316-0b9ef474905e-kube-api-access-bv2gl\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn\" (UID: \"af3d6e8a-e6de-4423-b316-0b9ef474905e\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:11:58.812076 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:58.812046 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:11:59.134425 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:59.134397 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn"] Oct 30 17:11:59.137195 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:11:59.137167 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3d6e8a_e6de_4423_b316_0b9ef474905e.slice/crio-60fb8bd1c8e060f2644e1dfe751d4139de2d03d2feecf0af5fb80fe7cb8f49bc WatchSource:0}: Error finding container 60fb8bd1c8e060f2644e1dfe751d4139de2d03d2feecf0af5fb80fe7cb8f49bc: Status 404 returned error can't find the container with id 60fb8bd1c8e060f2644e1dfe751d4139de2d03d2feecf0af5fb80fe7cb8f49bc Oct 30 17:11:59.772701 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:59.772660 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" event={"ID":"af3d6e8a-e6de-4423-b316-0b9ef474905e","Type":"ContainerStarted","Data":"4455c9e4aec786700d732f6f42bf78a4be6e0951cc0296597a7d84567242123f"} Oct 30 17:11:59.773655 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:59.773584 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:11:59.773655 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:59.773631 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" event={"ID":"af3d6e8a-e6de-4423-b316-0b9ef474905e","Type":"ContainerStarted","Data":"60fb8bd1c8e060f2644e1dfe751d4139de2d03d2feecf0af5fb80fe7cb8f49bc"} Oct 30 17:11:59.828111 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:11:59.828032 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" podStartSLOduration=1.828018696 podStartE2EDuration="1.828018696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:59.824381829 +0000 UTC m=+2950.167055867" watchObservedRunningTime="2026-10-30 17:11:59.828018696 +0000 UTC m=+2950.170692733" Oct 30 17:12:10.784847 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:10.784818 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:12:20.290460 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:12:20.290418 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:12:50.290619 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:12:50.290578 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:12:50.337765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:50.337716 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 17:12:50.366035 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:50.366012 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 17:12:51.167423 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:51.167386 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb"] Oct 30 17:12:51.172859 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:51.172827 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" Oct 30 17:12:51.187126 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:51.187100 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb"] Oct 30 17:12:51.234301 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:51.234272 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-24acb2\"" Oct 30 17:12:51.235307 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:51.235284 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-24acb2-dockercfg-t2lpr\"" Oct 30 17:12:51.403750 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:51.391853 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs8ls\" (UniqueName: \"kubernetes.io/projected/1f65a199-0838-4aa9-85d4-fb082370d14c-kube-api-access-cs8ls\") pod \"payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb\" (UID: \"1f65a199-0838-4aa9-85d4-fb082370d14c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" Oct 30 17:12:51.404852 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:51.404817 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f65a199-0838-4aa9-85d4-fb082370d14c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb\" (UID: \"1f65a199-0838-4aa9-85d4-fb082370d14c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" Oct 30 17:12:51.507238 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:51.507204 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f65a199-0838-4aa9-85d4-fb082370d14c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb\" (UID: \"1f65a199-0838-4aa9-85d4-fb082370d14c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" Oct 30 17:12:51.507451 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:51.507306 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8ls\" (UniqueName: \"kubernetes.io/projected/1f65a199-0838-4aa9-85d4-fb082370d14c-kube-api-access-cs8ls\") pod \"payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb\" (UID: \"1f65a199-0838-4aa9-85d4-fb082370d14c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" Oct 30 17:12:51.507994 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:51.507971 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f65a199-0838-4aa9-85d4-fb082370d14c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb\" (UID: \"1f65a199-0838-4aa9-85d4-fb082370d14c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" Oct 30 17:12:51.534077 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:51.534050 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs8ls\" (UniqueName: \"kubernetes.io/projected/1f65a199-0838-4aa9-85d4-fb082370d14c-kube-api-access-cs8ls\") pod \"payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb\" (UID: \"1f65a199-0838-4aa9-85d4-fb082370d14c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" Oct 30 17:12:51.814225 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:51.814142 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" Oct 30 17:12:52.129997 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:12:52.129966 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f65a199_0838_4aa9_85d4_fb082370d14c.slice/crio-49311fc336c11b18950f17c0f081bebcc4240ecd1a6e667d33a75c179e751073 WatchSource:0}: Error finding container 49311fc336c11b18950f17c0f081bebcc4240ecd1a6e667d33a75c179e751073: Status 404 returned error can't find the container with id 49311fc336c11b18950f17c0f081bebcc4240ecd1a6e667d33a75c179e751073 Oct 30 17:12:52.130859 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:52.130831 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb"] Oct 30 17:12:52.385223 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:52.385129 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" event={"ID":"1f65a199-0838-4aa9-85d4-fb082370d14c","Type":"ContainerStarted","Data":"f852edf94e0df1be64bb72d4eab661aa4a48dc0b3ced32280103e8f4a8abc2dd"} Oct 30 17:12:52.385223 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:52.385191 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" event={"ID":"1f65a199-0838-4aa9-85d4-fb082370d14c","Type":"ContainerStarted","Data":"49311fc336c11b18950f17c0f081bebcc4240ecd1a6e667d33a75c179e751073"} Oct 30 17:12:52.385223 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:52.385217 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" Oct 30 17:12:52.432972 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:12:52.432912 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" podStartSLOduration=1.43289896 podStartE2EDuration="1.43289896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:52.430665827 +0000 UTC m=+3002.773339879" watchObservedRunningTime="2026-10-30 17:12:52.43289896 +0000 UTC m=+3002.775572997" Oct 30 17:13:03.396077 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:03.396008 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" Oct 30 17:13:20.291189 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:13:20.291146 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:13:31.383760 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.383707 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk"] Oct 30 17:13:31.388431 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.388412 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:13:31.407776 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.407751 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk"] Oct 30 17:13:31.416844 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.416818 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-f94fb5\"/\"agc-cred-9ec3f2d49065\"" Oct 30 17:13:31.552075 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.552043 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrcj5\" (UniqueName: \"kubernetes.io/projected/a2e800b2-30e2-421e-a5db-9a16b6c346f6-kube-api-access-hrcj5\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk\" (UID: \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:13:31.552272 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.552252 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2e800b2-30e2-421e-a5db-9a16b6c346f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk\" (UID: \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:13:31.552380 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.552363 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2e800b2-30e2-421e-a5db-9a16b6c346f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk\" (UID: \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:13:31.552474 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.552449 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e800b2-30e2-421e-a5db-9a16b6c346f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk\" (UID: \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:13:31.653822 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.653735 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2e800b2-30e2-421e-a5db-9a16b6c346f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk\" (UID: \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:13:31.653970 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.653833 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2e800b2-30e2-421e-a5db-9a16b6c346f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk\" (UID: \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:13:31.653970 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.653868 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e800b2-30e2-421e-a5db-9a16b6c346f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk\" (UID: \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:13:31.653970 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.653944 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrcj5\" (UniqueName: \"kubernetes.io/projected/a2e800b2-30e2-421e-a5db-9a16b6c346f6-kube-api-access-hrcj5\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk\" (UID: \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:13:31.654410 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.654388 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2e800b2-30e2-421e-a5db-9a16b6c346f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk\" (UID: \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:13:31.654658 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.654637 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e800b2-30e2-421e-a5db-9a16b6c346f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk\" (UID: \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:13:31.656765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.656743 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2e800b2-30e2-421e-a5db-9a16b6c346f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk\" (UID: \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:13:31.679452 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.679421 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrcj5\" (UniqueName: \"kubernetes.io/projected/a2e800b2-30e2-421e-a5db-9a16b6c346f6-kube-api-access-hrcj5\") pod \"payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk\" (UID: \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\") " pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:13:31.703185 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:31.703162 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:13:32.037438 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:32.037406 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk"] Oct 30 17:13:32.037739 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:13:32.037702 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e800b2_30e2_421e_a5db_9a16b6c346f6.slice/crio-a1d49163aad88a84fe288b9858d826c307f68fc1354500af1dc7d50a52d1e846 WatchSource:0}: Error finding container a1d49163aad88a84fe288b9858d826c307f68fc1354500af1dc7d50a52d1e846: Status 404 returned error can't find the container with id a1d49163aad88a84fe288b9858d826c307f68fc1354500af1dc7d50a52d1e846 Oct 30 17:13:32.810994 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:32.810957 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" event={"ID":"a2e800b2-30e2-421e-a5db-9a16b6c346f6","Type":"ContainerStarted","Data":"304feb67af22bad1d1411ef5d00db789890fed0c5be514570f5275802fd0a46b"} Oct 30 17:13:32.811508 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:32.811023 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" event={"ID":"a2e800b2-30e2-421e-a5db-9a16b6c346f6","Type":"ContainerStarted","Data":"a1d49163aad88a84fe288b9858d826c307f68fc1354500af1dc7d50a52d1e846"} Oct 30 17:13:32.811508 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:32.811047 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:13:32.871753 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:32.871688 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" podStartSLOduration=1.871676914 podStartE2EDuration="1.871676914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:32.871412952 +0000 UTC m=+3043.214086989" watchObservedRunningTime="2026-10-30 17:13:32.871676914 +0000 UTC m=+3043.214350951" Oct 30 17:13:34.323597 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.323560 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52"] Oct 30 17:13:34.328979 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.328959 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:13:34.350301 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.350273 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52"] Oct 30 17:13:34.357320 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.357295 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-24acb2-serving-cert\"" Oct 30 17:13:34.408328 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.408289 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ed523f1-369d-4559-a4be-74633818bc17-maas-api-tls\") pod \"maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52\" (UID: \"8ed523f1-369d-4559-a4be-74633818bc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:13:34.408865 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.408821 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed523f1-369d-4559-a4be-74633818bc17-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52\" (UID: \"8ed523f1-369d-4559-a4be-74633818bc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:13:34.409011 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.408940 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ed523f1-369d-4559-a4be-74633818bc17-metrics-certs\") pod \"maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52\" (UID: \"8ed523f1-369d-4559-a4be-74633818bc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:13:34.409011 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.409002 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg4dl\" (UniqueName: \"kubernetes.io/projected/8ed523f1-369d-4559-a4be-74633818bc17-kube-api-access-mg4dl\") pod \"maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52\" (UID: \"8ed523f1-369d-4559-a4be-74633818bc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:13:34.509979 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.509943 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg4dl\" (UniqueName: \"kubernetes.io/projected/8ed523f1-369d-4559-a4be-74633818bc17-kube-api-access-mg4dl\") pod \"maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52\" (UID: \"8ed523f1-369d-4559-a4be-74633818bc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:13:34.510158 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.510069 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ed523f1-369d-4559-a4be-74633818bc17-maas-api-tls\") pod \"maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52\" (UID: \"8ed523f1-369d-4559-a4be-74633818bc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:13:34.510348 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.510327 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed523f1-369d-4559-a4be-74633818bc17-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52\" (UID: \"8ed523f1-369d-4559-a4be-74633818bc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:13:34.510418 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.510372 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ed523f1-369d-4559-a4be-74633818bc17-metrics-certs\") pod \"maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52\" (UID: \"8ed523f1-369d-4559-a4be-74633818bc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:13:34.511113 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.511089 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed523f1-369d-4559-a4be-74633818bc17-openshift-service-ca\") pod \"maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52\" (UID: \"8ed523f1-369d-4559-a4be-74633818bc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:13:34.513089 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.513068 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ed523f1-369d-4559-a4be-74633818bc17-metrics-certs\") pod \"maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52\" (UID: \"8ed523f1-369d-4559-a4be-74633818bc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:13:34.513342 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.513079 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ed523f1-369d-4559-a4be-74633818bc17-maas-api-tls\") pod \"maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52\" (UID: \"8ed523f1-369d-4559-a4be-74633818bc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:13:34.535440 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.535419 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg4dl\" (UniqueName: \"kubernetes.io/projected/8ed523f1-369d-4559-a4be-74633818bc17-kube-api-access-mg4dl\") pod \"maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52\" (UID: \"8ed523f1-369d-4559-a4be-74633818bc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:13:34.648128 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.648035 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:13:34.973871 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:13:34.973836 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed523f1_369d_4559_a4be_74633818bc17.slice/crio-cdc66c1ed69ab23e0299ceb5fa7d15326622aadecc635f3e19a1833ef56cde8a WatchSource:0}: Error finding container cdc66c1ed69ab23e0299ceb5fa7d15326622aadecc635f3e19a1833ef56cde8a: Status 404 returned error can't find the container with id cdc66c1ed69ab23e0299ceb5fa7d15326622aadecc635f3e19a1833ef56cde8a Oct 30 17:13:34.974998 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.974977 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52"] Oct 30 17:13:34.975785 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:34.975766 4861 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:13:35.835172 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:35.835135 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" event={"ID":"8ed523f1-369d-4559-a4be-74633818bc17","Type":"ContainerStarted","Data":"3dbe1f69681558ccdd47418ec8d69a1e6cd7f54174669e602e3236ff152d5695"} Oct 30 17:13:35.835607 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:35.835196 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" event={"ID":"8ed523f1-369d-4559-a4be-74633818bc17","Type":"ContainerStarted","Data":"cdc66c1ed69ab23e0299ceb5fa7d15326622aadecc635f3e19a1833ef56cde8a"} Oct 30 17:13:35.835607 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:35.835352 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:13:35.895159 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:35.895111 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" podStartSLOduration=1.517397072 podStartE2EDuration="1.895097584s" totalImagesPullingTime="377.700512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:35.894547266 +0000 UTC m=+3046.237221303" watchObservedRunningTime="2026-10-30 17:13:35.895097584 +0000 UTC m=+3046.237771651" Oct 30 17:13:39.540757 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:39.540709 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn"] Oct 30 17:13:41.849254 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:41.849224 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:13:42.488155 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:42.488118 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4"] Oct 30 17:13:43.821998 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:43.821967 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:13:43.902679 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:43.902646 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw"] Oct 30 17:13:44.541247 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:44.541210 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" podUID="af3d6e8a-e6de-4423-b316-0b9ef474905e" containerName="payload-processing" containerID="cri-o://4455c9e4aec786700d732f6f42bf78a4be6e0951cc0296597a7d84567242123f" gracePeriod=25 Oct 30 17:13:44.845221 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:44.845188 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:13:44.921057 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:44.921027 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af3d6e8a-e6de-4423-b316-0b9ef474905e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/af3d6e8a-e6de-4423-b316-0b9ef474905e-provider-credentials\") pod \"af3d6e8a-e6de-4423-b316-0b9ef474905e\" (UID: \"af3d6e8a-e6de-4423-b316-0b9ef474905e\") " Oct 30 17:13:44.921236 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:44.921185 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af3d6e8a-e6de-4423-b316-0b9ef474905e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3d6e8a-e6de-4423-b316-0b9ef474905e-plugins-config-volume\") pod \"af3d6e8a-e6de-4423-b316-0b9ef474905e\" (UID: \"af3d6e8a-e6de-4423-b316-0b9ef474905e\") " Oct 30 17:13:44.921307 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:44.921278 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af3d6e8a-e6de-4423-b316-0b9ef474905e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af3d6e8a-e6de-4423-b316-0b9ef474905e-routing-overlay-volume\") pod \"af3d6e8a-e6de-4423-b316-0b9ef474905e\" (UID: \"af3d6e8a-e6de-4423-b316-0b9ef474905e\") " Oct 30 17:13:44.921482 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:44.921459 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af3d6e8a-e6de-4423-b316-0b9ef474905e-kube-api-access-bv2gl\" (UniqueName: \"kubernetes.io/projected/af3d6e8a-e6de-4423-b316-0b9ef474905e-kube-api-access-bv2gl\") pod \"af3d6e8a-e6de-4423-b316-0b9ef474905e\" (UID: \"af3d6e8a-e6de-4423-b316-0b9ef474905e\") " Oct 30 17:13:44.921651 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:44.921625 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3d6e8a-e6de-4423-b316-0b9ef474905e-plugins-config-volume" pod "af3d6e8a-e6de-4423-b316-0b9ef474905e" (UID: "af3d6e8a-e6de-4423-b316-0b9ef474905e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:44.921651 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:44.921635 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af3d6e8a-e6de-4423-b316-0b9ef474905e-routing-overlay-volume" pod "af3d6e8a-e6de-4423-b316-0b9ef474905e" (UID: "af3d6e8a-e6de-4423-b316-0b9ef474905e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:44.922642 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:44.922620 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af3d6e8a-e6de-4423-b316-0b9ef474905e-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:44.922768 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:44.922646 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af3d6e8a-e6de-4423-b316-0b9ef474905e-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:44.923662 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:44.923637 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3d6e8a-e6de-4423-b316-0b9ef474905e-provider-credentials" pod "af3d6e8a-e6de-4423-b316-0b9ef474905e" (UID: "af3d6e8a-e6de-4423-b316-0b9ef474905e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:44.925100 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:44.925071 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af3d6e8a-e6de-4423-b316-0b9ef474905e-kube-api-access-bv2gl" pod "af3d6e8a-e6de-4423-b316-0b9ef474905e" (UID: "af3d6e8a-e6de-4423-b316-0b9ef474905e"). InnerVolumeSpecName "kube-api-access-bv2gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:45.024103 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:45.024077 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv2gl\" (UniqueName: \"kubernetes.io/projected/af3d6e8a-e6de-4423-b316-0b9ef474905e-kube-api-access-bv2gl\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:45.024103 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:45.024100 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af3d6e8a-e6de-4423-b316-0b9ef474905e-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:45.055434 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:45.055412 4861 generic.go:356] "Generic (PLEG): container finished" podID="af3d6e8a-e6de-4423-b316-0b9ef474905e" containerID="4455c9e4aec786700d732f6f42bf78a4be6e0951cc0296597a7d84567242123f" exitCode=0 Oct 30 17:13:45.055548 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:45.055457 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" Oct 30 17:13:45.055548 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:45.055494 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" event={"ID":"af3d6e8a-e6de-4423-b316-0b9ef474905e","Type":"ContainerDied","Data":"4455c9e4aec786700d732f6f42bf78a4be6e0951cc0296597a7d84567242123f"} Oct 30 17:13:45.055548 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:45.055542 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn" event={"ID":"af3d6e8a-e6de-4423-b316-0b9ef474905e","Type":"ContainerDied","Data":"60fb8bd1c8e060f2644e1dfe751d4139de2d03d2feecf0af5fb80fe7cb8f49bc"} Oct 30 17:13:45.055684 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:45.055565 4861 scope.go:122] "RemoveContainer" containerID="4455c9e4aec786700d732f6f42bf78a4be6e0951cc0296597a7d84567242123f" Oct 30 17:13:45.067488 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:45.067469 4861 scope.go:122] "RemoveContainer" containerID="4455c9e4aec786700d732f6f42bf78a4be6e0951cc0296597a7d84567242123f" Oct 30 17:13:45.067804 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:13:45.067778 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4455c9e4aec786700d732f6f42bf78a4be6e0951cc0296597a7d84567242123f\": container with ID starting with 4455c9e4aec786700d732f6f42bf78a4be6e0951cc0296597a7d84567242123f not found: ID does not exist" containerID="4455c9e4aec786700d732f6f42bf78a4be6e0951cc0296597a7d84567242123f" Oct 30 17:13:45.067885 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:45.067817 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4455c9e4aec786700d732f6f42bf78a4be6e0951cc0296597a7d84567242123f"} err="failed to get container status \"4455c9e4aec786700d732f6f42bf78a4be6e0951cc0296597a7d84567242123f\": rpc error: code = NotFound desc = could not find container \"4455c9e4aec786700d732f6f42bf78a4be6e0951cc0296597a7d84567242123f\": container with ID starting with 4455c9e4aec786700d732f6f42bf78a4be6e0951cc0296597a7d84567242123f not found: ID does not exist" Oct 30 17:13:45.114668 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:45.114646 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn"] Oct 30 17:13:45.132131 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:45.132101 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-7cc8p72cn"] Oct 30 17:13:45.810092 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:45.810058 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af3d6e8a-e6de-4423-b316-0b9ef474905e" path="/var/lib/kubelet/pods/af3d6e8a-e6de-4423-b316-0b9ef474905e/volumes" Oct 30 17:13:47.506747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:47.499861 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" podUID="c2ec4f78-379e-4f7b-9b08-686be0d0524a" containerName="payload-processing" containerID="cri-o://85ba0434e882099789468ca6e8bcf12e64c9e00fd98fb195decde1b9d3eedcee" gracePeriod=25 Oct 30 17:13:47.611619 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:13:47.611594 4861 logging.go:55] [core] [Channel #14050 SubChannel #14051]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.7:50052", ServerName: "10.132.1.7:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.7:50052: connect: connection refused" Oct 30 17:13:47.801936 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:47.801905 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:13:47.886514 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:47.886477 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ec4f78-379e-4f7b-9b08-686be0d0524a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2ec4f78-379e-4f7b-9b08-686be0d0524a-provider-credentials\") pod \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\" (UID: \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\") " Oct 30 17:13:47.886696 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:47.886658 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2ec4f78-379e-4f7b-9b08-686be0d0524a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ec4f78-379e-4f7b-9b08-686be0d0524a-plugins-config-volume\") pod \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\" (UID: \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\") " Oct 30 17:13:47.886850 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:47.886830 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ec4f78-379e-4f7b-9b08-686be0d0524a-kube-api-access-kglkn\" (UniqueName: \"kubernetes.io/projected/c2ec4f78-379e-4f7b-9b08-686be0d0524a-kube-api-access-kglkn\") pod \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\" (UID: \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\") " Oct 30 17:13:47.886923 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:47.886908 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2ec4f78-379e-4f7b-9b08-686be0d0524a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2ec4f78-379e-4f7b-9b08-686be0d0524a-routing-overlay-volume\") pod \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\" (UID: \"c2ec4f78-379e-4f7b-9b08-686be0d0524a\") " Oct 30 17:13:47.887172 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:47.887148 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ec4f78-379e-4f7b-9b08-686be0d0524a-plugins-config-volume" pod "c2ec4f78-379e-4f7b-9b08-686be0d0524a" (UID: "c2ec4f78-379e-4f7b-9b08-686be0d0524a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:47.888274 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:47.887622 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ec4f78-379e-4f7b-9b08-686be0d0524a-routing-overlay-volume" pod "c2ec4f78-379e-4f7b-9b08-686be0d0524a" (UID: "c2ec4f78-379e-4f7b-9b08-686be0d0524a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:47.888274 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:47.887904 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2ec4f78-379e-4f7b-9b08-686be0d0524a-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.888274 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:47.887926 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ec4f78-379e-4f7b-9b08-686be0d0524a-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.890020 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:47.889984 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ec4f78-379e-4f7b-9b08-686be0d0524a-kube-api-access-kglkn" pod "c2ec4f78-379e-4f7b-9b08-686be0d0524a" (UID: "c2ec4f78-379e-4f7b-9b08-686be0d0524a"). InnerVolumeSpecName "kube-api-access-kglkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:47.890685 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:47.890657 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ec4f78-379e-4f7b-9b08-686be0d0524a-provider-credentials" pod "c2ec4f78-379e-4f7b-9b08-686be0d0524a" (UID: "c2ec4f78-379e-4f7b-9b08-686be0d0524a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:47.989507 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:47.989469 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kglkn\" (UniqueName: \"kubernetes.io/projected/c2ec4f78-379e-4f7b-9b08-686be0d0524a-kube-api-access-kglkn\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:47.989507 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:47.989501 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2ec4f78-379e-4f7b-9b08-686be0d0524a-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:48.084215 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:48.084135 4861 generic.go:356] "Generic (PLEG): container finished" podID="c2ec4f78-379e-4f7b-9b08-686be0d0524a" containerID="85ba0434e882099789468ca6e8bcf12e64c9e00fd98fb195decde1b9d3eedcee" exitCode=0 Oct 30 17:13:48.084215 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:48.084204 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" Oct 30 17:13:48.084452 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:48.084230 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" event={"ID":"c2ec4f78-379e-4f7b-9b08-686be0d0524a","Type":"ContainerDied","Data":"85ba0434e882099789468ca6e8bcf12e64c9e00fd98fb195decde1b9d3eedcee"} Oct 30 17:13:48.084452 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:48.084290 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" event={"ID":"c2ec4f78-379e-4f7b-9b08-686be0d0524a","Type":"ContainerDied","Data":"5a507575831cef08bffdf6e74d322f32d4ec758ab8b65ad04fa7b414b124860d"} Oct 30 17:13:48.084452 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:48.084319 4861 scope.go:122] "RemoveContainer" containerID="85ba0434e882099789468ca6e8bcf12e64c9e00fd98fb195decde1b9d3eedcee" Oct 30 17:13:48.099600 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:48.099574 4861 scope.go:122] "RemoveContainer" containerID="85ba0434e882099789468ca6e8bcf12e64c9e00fd98fb195decde1b9d3eedcee" Oct 30 17:13:48.099961 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:13:48.099938 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ba0434e882099789468ca6e8bcf12e64c9e00fd98fb195decde1b9d3eedcee\": container with ID starting with 85ba0434e882099789468ca6e8bcf12e64c9e00fd98fb195decde1b9d3eedcee not found: ID does not exist" containerID="85ba0434e882099789468ca6e8bcf12e64c9e00fd98fb195decde1b9d3eedcee" Oct 30 17:13:48.100055 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:48.099968 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ba0434e882099789468ca6e8bcf12e64c9e00fd98fb195decde1b9d3eedcee"} err="failed to get container status \"85ba0434e882099789468ca6e8bcf12e64c9e00fd98fb195decde1b9d3eedcee\": rpc error: code = NotFound desc = could not find container \"85ba0434e882099789468ca6e8bcf12e64c9e00fd98fb195decde1b9d3eedcee\": container with ID starting with 85ba0434e882099789468ca6e8bcf12e64c9e00fd98fb195decde1b9d3eedcee not found: ID does not exist" Oct 30 17:13:48.143691 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:48.143667 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4"] Oct 30 17:13:48.157364 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:48.157342 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4"] Oct 30 17:13:48.611992 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:48.611943 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-13c495/payload-processing-external-model-e2e-extmod-b-13c495-6bbdw77v4" podUID="c2ec4f78-379e-4f7b-9b08-686be0d0524a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.7:50052\" within 1s: context deadline exceeded" Oct 30 17:13:48.611992 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:13:48.611986 4861 logging.go:55] [core] [Channel #14050 SubChannel #14051]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.7:50052", ServerName: "10.132.1.7:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.7:50052: operation was canceled" Oct 30 17:13:48.903806 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:48.903699 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" podUID="06ac0898-f2b4-4605-9f7b-2a1dd4628995" containerName="payload-processing" containerID="cri-o://4e25a247ff70f49bad9eee07460f0b3a35b65bb1a5d3418b1f5bf6a8352ab816" gracePeriod=25 Oct 30 17:13:49.104746 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:49.102588 4861 generic.go:356] "Generic (PLEG): container finished" podID="06ac0898-f2b4-4605-9f7b-2a1dd4628995" containerID="4e25a247ff70f49bad9eee07460f0b3a35b65bb1a5d3418b1f5bf6a8352ab816" exitCode=0 Oct 30 17:13:49.104746 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:49.102599 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" event={"ID":"06ac0898-f2b4-4605-9f7b-2a1dd4628995","Type":"ContainerDied","Data":"4e25a247ff70f49bad9eee07460f0b3a35b65bb1a5d3418b1f5bf6a8352ab816"} Oct 30 17:13:49.227533 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:49.227508 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:13:49.307956 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:49.307927 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ac0898-f2b4-4605-9f7b-2a1dd4628995-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ac0898-f2b4-4605-9f7b-2a1dd4628995-plugins-config-volume\") pod \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\" (UID: \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\") " Oct 30 17:13:49.308125 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:49.308000 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ac0898-f2b4-4605-9f7b-2a1dd4628995-provider-credentials\" (UniqueName: \"kubernetes.io/projected/06ac0898-f2b4-4605-9f7b-2a1dd4628995-provider-credentials\") pod \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\" (UID: \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\") " Oct 30 17:13:49.308125 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:49.308107 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ac0898-f2b4-4605-9f7b-2a1dd4628995-kube-api-access-9fj66\" (UniqueName: \"kubernetes.io/projected/06ac0898-f2b4-4605-9f7b-2a1dd4628995-kube-api-access-9fj66\") pod \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\" (UID: \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\") " Oct 30 17:13:49.308225 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:49.308210 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ac0898-f2b4-4605-9f7b-2a1dd4628995-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06ac0898-f2b4-4605-9f7b-2a1dd4628995-routing-overlay-volume\") pod \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\" (UID: \"06ac0898-f2b4-4605-9f7b-2a1dd4628995\") " Oct 30 17:13:49.308339 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:49.308313 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ac0898-f2b4-4605-9f7b-2a1dd4628995-plugins-config-volume" pod "06ac0898-f2b4-4605-9f7b-2a1dd4628995" (UID: "06ac0898-f2b4-4605-9f7b-2a1dd4628995"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:49.308627 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:49.308597 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ac0898-f2b4-4605-9f7b-2a1dd4628995-routing-overlay-volume" pod "06ac0898-f2b4-4605-9f7b-2a1dd4628995" (UID: "06ac0898-f2b4-4605-9f7b-2a1dd4628995"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:49.308983 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:49.308961 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ac0898-f2b4-4605-9f7b-2a1dd4628995-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:49.308983 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:49.308984 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06ac0898-f2b4-4605-9f7b-2a1dd4628995-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:49.310541 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:49.310513 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ac0898-f2b4-4605-9f7b-2a1dd4628995-provider-credentials" pod "06ac0898-f2b4-4605-9f7b-2a1dd4628995" (UID: "06ac0898-f2b4-4605-9f7b-2a1dd4628995"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:49.310636 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:49.310566 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ac0898-f2b4-4605-9f7b-2a1dd4628995-kube-api-access-9fj66" pod "06ac0898-f2b4-4605-9f7b-2a1dd4628995" (UID: "06ac0898-f2b4-4605-9f7b-2a1dd4628995"). InnerVolumeSpecName "kube-api-access-9fj66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:49.423771 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:49.414915 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06ac0898-f2b4-4605-9f7b-2a1dd4628995-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:49.423771 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:49.414950 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fj66\" (UniqueName: \"kubernetes.io/projected/06ac0898-f2b4-4605-9f7b-2a1dd4628995-kube-api-access-9fj66\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:49.821168 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:49.821137 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2ec4f78-379e-4f7b-9b08-686be0d0524a" path="/var/lib/kubelet/pods/c2ec4f78-379e-4f7b-9b08-686be0d0524a/volumes" Oct 30 17:13:50.115037 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:50.114943 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" Oct 30 17:13:50.115222 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:50.114948 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw" event={"ID":"06ac0898-f2b4-4605-9f7b-2a1dd4628995","Type":"ContainerDied","Data":"0caaf8327f76b54b958a0ed80d54c24708db1eeb5d318fff88cb630ede743a53"} Oct 30 17:13:50.115222 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:50.115085 4861 scope.go:122] "RemoveContainer" containerID="4e25a247ff70f49bad9eee07460f0b3a35b65bb1a5d3418b1f5bf6a8352ab816" Oct 30 17:13:50.173711 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:50.173678 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw"] Oct 30 17:13:50.187543 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:50.187518 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-5649589lw"] Oct 30 17:13:50.292602 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:13:50.292562 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:13:50.420905 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:50.420832 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk"] Oct 30 17:13:51.810539 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:51.810501 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ac0898-f2b4-4605-9f7b-2a1dd4628995" path="/var/lib/kubelet/pods/06ac0898-f2b4-4605-9f7b-2a1dd4628995/volumes" Oct 30 17:13:53.394384 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:53.394343 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4"] Oct 30 17:13:54.339567 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:54.339529 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc"] Oct 30 17:13:55.421566 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:55.421496 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" podUID="c263ff4a-fa28-4eaa-9d93-75a2923f9161" containerName="payload-processing" containerID="cri-o://4988e924f8095dbcd8b9e794ae2f2636cbfe08d3e4c1da9e44e03344aa0f36c2" gracePeriod=25 Oct 30 17:13:55.775027 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:55.774988 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:13:55.813484 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:55.813458 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c263ff4a-fa28-4eaa-9d93-75a2923f9161-kube-api-access-2tmzx\" (UniqueName: \"kubernetes.io/projected/c263ff4a-fa28-4eaa-9d93-75a2923f9161-kube-api-access-2tmzx\") pod \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\" (UID: \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\") " Oct 30 17:13:55.813631 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:55.813516 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c263ff4a-fa28-4eaa-9d93-75a2923f9161-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c263ff4a-fa28-4eaa-9d93-75a2923f9161-provider-credentials\") pod \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\" (UID: \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\") " Oct 30 17:13:55.813631 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:55.813549 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c263ff4a-fa28-4eaa-9d93-75a2923f9161-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c263ff4a-fa28-4eaa-9d93-75a2923f9161-routing-overlay-volume\") pod \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\" (UID: \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\") " Oct 30 17:13:55.813631 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:55.813589 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c263ff4a-fa28-4eaa-9d93-75a2923f9161-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c263ff4a-fa28-4eaa-9d93-75a2923f9161-plugins-config-volume\") pod \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\" (UID: \"c263ff4a-fa28-4eaa-9d93-75a2923f9161\") " Oct 30 17:13:55.813956 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:55.813916 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c263ff4a-fa28-4eaa-9d93-75a2923f9161-routing-overlay-volume" pod "c263ff4a-fa28-4eaa-9d93-75a2923f9161" (UID: "c263ff4a-fa28-4eaa-9d93-75a2923f9161"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:55.814081 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:55.814018 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c263ff4a-fa28-4eaa-9d93-75a2923f9161-plugins-config-volume" pod "c263ff4a-fa28-4eaa-9d93-75a2923f9161" (UID: "c263ff4a-fa28-4eaa-9d93-75a2923f9161"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:55.814298 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:55.814275 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c263ff4a-fa28-4eaa-9d93-75a2923f9161-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:55.814387 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:55.814300 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c263ff4a-fa28-4eaa-9d93-75a2923f9161-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:55.815919 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:55.815897 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c263ff4a-fa28-4eaa-9d93-75a2923f9161-provider-credentials" pod "c263ff4a-fa28-4eaa-9d93-75a2923f9161" (UID: "c263ff4a-fa28-4eaa-9d93-75a2923f9161"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:55.816109 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:55.816091 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c263ff4a-fa28-4eaa-9d93-75a2923f9161-kube-api-access-2tmzx" pod "c263ff4a-fa28-4eaa-9d93-75a2923f9161" (UID: "c263ff4a-fa28-4eaa-9d93-75a2923f9161"). InnerVolumeSpecName "kube-api-access-2tmzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:55.916368 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:55.916340 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tmzx\" (UniqueName: \"kubernetes.io/projected/c263ff4a-fa28-4eaa-9d93-75a2923f9161-kube-api-access-2tmzx\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:55.916368 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:55.916368 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c263ff4a-fa28-4eaa-9d93-75a2923f9161-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:56.229937 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:56.229902 4861 generic.go:356] "Generic (PLEG): container finished" podID="c263ff4a-fa28-4eaa-9d93-75a2923f9161" containerID="4988e924f8095dbcd8b9e794ae2f2636cbfe08d3e4c1da9e44e03344aa0f36c2" exitCode=0 Oct 30 17:13:56.230148 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:56.229969 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" Oct 30 17:13:56.230148 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:56.229975 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" event={"ID":"c263ff4a-fa28-4eaa-9d93-75a2923f9161","Type":"ContainerDied","Data":"4988e924f8095dbcd8b9e794ae2f2636cbfe08d3e4c1da9e44e03344aa0f36c2"} Oct 30 17:13:56.230148 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:56.230034 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk" event={"ID":"c263ff4a-fa28-4eaa-9d93-75a2923f9161","Type":"ContainerDied","Data":"52476be5a7a12b6eeefdf599583805526fcec30acaaf3dbb8c61168ff8555c56"} Oct 30 17:13:56.230148 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:56.230063 4861 scope.go:122] "RemoveContainer" containerID="4988e924f8095dbcd8b9e794ae2f2636cbfe08d3e4c1da9e44e03344aa0f36c2" Oct 30 17:13:56.242246 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:56.242224 4861 scope.go:122] "RemoveContainer" containerID="4988e924f8095dbcd8b9e794ae2f2636cbfe08d3e4c1da9e44e03344aa0f36c2" Oct 30 17:13:56.242521 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:13:56.242498 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4988e924f8095dbcd8b9e794ae2f2636cbfe08d3e4c1da9e44e03344aa0f36c2\": container with ID starting with 4988e924f8095dbcd8b9e794ae2f2636cbfe08d3e4c1da9e44e03344aa0f36c2 not found: ID does not exist" containerID="4988e924f8095dbcd8b9e794ae2f2636cbfe08d3e4c1da9e44e03344aa0f36c2" Oct 30 17:13:56.242621 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:56.242527 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4988e924f8095dbcd8b9e794ae2f2636cbfe08d3e4c1da9e44e03344aa0f36c2"} err="failed to get container status \"4988e924f8095dbcd8b9e794ae2f2636cbfe08d3e4c1da9e44e03344aa0f36c2\": rpc error: code = NotFound desc = could not find container \"4988e924f8095dbcd8b9e794ae2f2636cbfe08d3e4c1da9e44e03344aa0f36c2\": container with ID starting with 4988e924f8095dbcd8b9e794ae2f2636cbfe08d3e4c1da9e44e03344aa0f36c2 not found: ID does not exist" Oct 30 17:13:56.289425 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:56.289398 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk"] Oct 30 17:13:56.302874 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:56.302853 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-43197b/payload-processing-external-model-e2e-extmod-b-43197b-6b8dx2vpk"] Oct 30 17:13:57.809967 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:57.809934 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c263ff4a-fa28-4eaa-9d93-75a2923f9161" path="/var/lib/kubelet/pods/c263ff4a-fa28-4eaa-9d93-75a2923f9161/volumes" Oct 30 17:13:58.394870 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:58.394830 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" podUID="978ef4f2-ef94-42ed-abcc-01f26e1e5215" containerName="payload-processing" containerID="cri-o://b0690191ff842d3f33ea56db6c9515e4131adbc0b0038a2255bcca4849d7c3a9" gracePeriod=25 Oct 30 17:13:58.684876 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:58.684844 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:13:58.798548 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:58.798519 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978ef4f2-ef94-42ed-abcc-01f26e1e5215-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/978ef4f2-ef94-42ed-abcc-01f26e1e5215-routing-overlay-volume\") pod \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\" (UID: \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\") " Oct 30 17:13:58.798743 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:58.798710 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978ef4f2-ef94-42ed-abcc-01f26e1e5215-kube-api-access-lm7zx\" (UniqueName: \"kubernetes.io/projected/978ef4f2-ef94-42ed-abcc-01f26e1e5215-kube-api-access-lm7zx\") pod \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\" (UID: \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\") " Oct 30 17:13:58.798813 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:58.798770 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/978ef4f2-ef94-42ed-abcc-01f26e1e5215-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978ef4f2-ef94-42ed-abcc-01f26e1e5215-plugins-config-volume\") pod \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\" (UID: \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\") " Oct 30 17:13:58.798862 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:58.798824 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/978ef4f2-ef94-42ed-abcc-01f26e1e5215-provider-credentials\" (UniqueName: \"kubernetes.io/projected/978ef4f2-ef94-42ed-abcc-01f26e1e5215-provider-credentials\") pod \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\" (UID: \"978ef4f2-ef94-42ed-abcc-01f26e1e5215\") " Oct 30 17:13:58.798956 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:58.798929 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978ef4f2-ef94-42ed-abcc-01f26e1e5215-routing-overlay-volume" pod "978ef4f2-ef94-42ed-abcc-01f26e1e5215" (UID: "978ef4f2-ef94-42ed-abcc-01f26e1e5215"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:58.799372 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:58.799254 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/978ef4f2-ef94-42ed-abcc-01f26e1e5215-plugins-config-volume" pod "978ef4f2-ef94-42ed-abcc-01f26e1e5215" (UID: "978ef4f2-ef94-42ed-abcc-01f26e1e5215"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:58.800594 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:58.799811 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/978ef4f2-ef94-42ed-abcc-01f26e1e5215-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:58.800594 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:58.799840 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/978ef4f2-ef94-42ed-abcc-01f26e1e5215-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:58.802927 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:58.802900 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978ef4f2-ef94-42ed-abcc-01f26e1e5215-provider-credentials" pod "978ef4f2-ef94-42ed-abcc-01f26e1e5215" (UID: "978ef4f2-ef94-42ed-abcc-01f26e1e5215"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:58.803038 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:58.802939 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/978ef4f2-ef94-42ed-abcc-01f26e1e5215-kube-api-access-lm7zx" pod "978ef4f2-ef94-42ed-abcc-01f26e1e5215" (UID: "978ef4f2-ef94-42ed-abcc-01f26e1e5215"). InnerVolumeSpecName "kube-api-access-lm7zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:58.901426 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:58.901325 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm7zx\" (UniqueName: \"kubernetes.io/projected/978ef4f2-ef94-42ed-abcc-01f26e1e5215-kube-api-access-lm7zx\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:58.901426 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:58.901364 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/978ef4f2-ef94-42ed-abcc-01f26e1e5215-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:59.298604 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.298562 4861 generic.go:356] "Generic (PLEG): container finished" podID="978ef4f2-ef94-42ed-abcc-01f26e1e5215" containerID="b0690191ff842d3f33ea56db6c9515e4131adbc0b0038a2255bcca4849d7c3a9" exitCode=0 Oct 30 17:13:59.298853 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.298620 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" Oct 30 17:13:59.298853 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.298626 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" event={"ID":"978ef4f2-ef94-42ed-abcc-01f26e1e5215","Type":"ContainerDied","Data":"b0690191ff842d3f33ea56db6c9515e4131adbc0b0038a2255bcca4849d7c3a9"} Oct 30 17:13:59.298853 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.298696 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4" event={"ID":"978ef4f2-ef94-42ed-abcc-01f26e1e5215","Type":"ContainerDied","Data":"67ab508aafb53ce8cf589c3582aab5791b82ec9cb7335da4860223ad53298a80"} Oct 30 17:13:59.298853 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.298763 4861 scope.go:122] "RemoveContainer" containerID="b0690191ff842d3f33ea56db6c9515e4131adbc0b0038a2255bcca4849d7c3a9" Oct 30 17:13:59.310915 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.310897 4861 scope.go:122] "RemoveContainer" containerID="b0690191ff842d3f33ea56db6c9515e4131adbc0b0038a2255bcca4849d7c3a9" Oct 30 17:13:59.311218 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:13:59.311189 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0690191ff842d3f33ea56db6c9515e4131adbc0b0038a2255bcca4849d7c3a9\": container with ID starting with b0690191ff842d3f33ea56db6c9515e4131adbc0b0038a2255bcca4849d7c3a9 not found: ID does not exist" containerID="b0690191ff842d3f33ea56db6c9515e4131adbc0b0038a2255bcca4849d7c3a9" Oct 30 17:13:59.311278 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.311227 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0690191ff842d3f33ea56db6c9515e4131adbc0b0038a2255bcca4849d7c3a9"} err="failed to get container status \"b0690191ff842d3f33ea56db6c9515e4131adbc0b0038a2255bcca4849d7c3a9\": rpc error: code = NotFound desc = could not find container \"b0690191ff842d3f33ea56db6c9515e4131adbc0b0038a2255bcca4849d7c3a9\": container with ID starting with b0690191ff842d3f33ea56db6c9515e4131adbc0b0038a2255bcca4849d7c3a9 not found: ID does not exist" Oct 30 17:13:59.340573 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.340547 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" podUID="f4c05e79-f99c-4b1b-8a48-3e366b6f5472" containerName="payload-processing" containerID="cri-o://18d264f58dbf569d3571194d4bce0027e584894b63e992c3b72be94bfb7c91be" gracePeriod=25 Oct 30 17:13:59.358035 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.358013 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4"] Oct 30 17:13:59.362299 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.362274 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6557jv4l4"] Oct 30 17:13:59.624225 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.624195 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:13:59.715401 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.715374 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-provider-credentials\") pod \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\" (UID: \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\") " Oct 30 17:13:59.715587 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.715503 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-routing-overlay-volume\") pod \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\" (UID: \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\") " Oct 30 17:13:59.715672 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.715597 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-kube-api-access-bxh42\" (UniqueName: \"kubernetes.io/projected/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-kube-api-access-bxh42\") pod \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\" (UID: \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\") " Oct 30 17:13:59.715672 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.715655 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-plugins-config-volume\") pod \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\" (UID: \"f4c05e79-f99c-4b1b-8a48-3e366b6f5472\") " Oct 30 17:13:59.715940 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.715915 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-routing-overlay-volume" pod "f4c05e79-f99c-4b1b-8a48-3e366b6f5472" (UID: "f4c05e79-f99c-4b1b-8a48-3e366b6f5472"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:59.716240 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.716210 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-plugins-config-volume" pod "f4c05e79-f99c-4b1b-8a48-3e366b6f5472" (UID: "f4c05e79-f99c-4b1b-8a48-3e366b6f5472"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:59.716431 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.716410 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:59.716541 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.716436 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:59.718548 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.718518 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-provider-credentials" pod "f4c05e79-f99c-4b1b-8a48-3e366b6f5472" (UID: "f4c05e79-f99c-4b1b-8a48-3e366b6f5472"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:59.718653 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.718613 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-kube-api-access-bxh42" pod "f4c05e79-f99c-4b1b-8a48-3e366b6f5472" (UID: "f4c05e79-f99c-4b1b-8a48-3e366b6f5472"). InnerVolumeSpecName "kube-api-access-bxh42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:59.842750 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.832345 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxh42\" (UniqueName: \"kubernetes.io/projected/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-kube-api-access-bxh42\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:59.842750 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.842584 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4c05e79-f99c-4b1b-8a48-3e366b6f5472-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:13:59.871842 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:13:59.871808 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="978ef4f2-ef94-42ed-abcc-01f26e1e5215" path="/var/lib/kubelet/pods/978ef4f2-ef94-42ed-abcc-01f26e1e5215/volumes" Oct 30 17:14:00.308112 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.308077 4861 generic.go:356] "Generic (PLEG): container finished" podID="f4c05e79-f99c-4b1b-8a48-3e366b6f5472" containerID="18d264f58dbf569d3571194d4bce0027e584894b63e992c3b72be94bfb7c91be" exitCode=0 Oct 30 17:14:00.308516 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.308121 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" Oct 30 17:14:00.308516 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.308161 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" event={"ID":"f4c05e79-f99c-4b1b-8a48-3e366b6f5472","Type":"ContainerDied","Data":"18d264f58dbf569d3571194d4bce0027e584894b63e992c3b72be94bfb7c91be"} Oct 30 17:14:00.308516 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.308212 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc" event={"ID":"f4c05e79-f99c-4b1b-8a48-3e366b6f5472","Type":"ContainerDied","Data":"fdd0edaa11fafad0e20e46b3af8e431a022e6666385a488e8cc1090d5df8fe5c"} Oct 30 17:14:00.308516 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.308234 4861 scope.go:122] "RemoveContainer" containerID="18d264f58dbf569d3571194d4bce0027e584894b63e992c3b72be94bfb7c91be" Oct 30 17:14:00.322471 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.322454 4861 scope.go:122] "RemoveContainer" containerID="18d264f58dbf569d3571194d4bce0027e584894b63e992c3b72be94bfb7c91be" Oct 30 17:14:00.322822 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:14:00.322800 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d264f58dbf569d3571194d4bce0027e584894b63e992c3b72be94bfb7c91be\": container with ID starting with 18d264f58dbf569d3571194d4bce0027e584894b63e992c3b72be94bfb7c91be not found: ID does not exist" containerID="18d264f58dbf569d3571194d4bce0027e584894b63e992c3b72be94bfb7c91be" Oct 30 17:14:00.322911 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.322831 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d264f58dbf569d3571194d4bce0027e584894b63e992c3b72be94bfb7c91be"} err="failed to get container status \"18d264f58dbf569d3571194d4bce0027e584894b63e992c3b72be94bfb7c91be\": rpc error: code = NotFound desc = could not find container \"18d264f58dbf569d3571194d4bce0027e584894b63e992c3b72be94bfb7c91be\": container with ID starting with 18d264f58dbf569d3571194d4bce0027e584894b63e992c3b72be94bfb7c91be not found: ID does not exist" Oct 30 17:14:00.373531 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.373499 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc"] Oct 30 17:14:00.386344 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.386318 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5758b7/payload-processing-external-model-e2e-extmod-b-5758b7-6c68zrwhc"] Oct 30 17:14:00.943295 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.943263 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr"] Oct 30 17:14:00.944216 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944191 4861 cpu_manager.go:414] "containerMap: removing container" podUID="f4c05e79-f99c-4b1b-8a48-3e366b6f5472" containerName="payload-processing" Oct 30 17:14:00.944216 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944215 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4c05e79-f99c-4b1b-8a48-3e366b6f5472" containerName="payload-processing" Oct 30 17:14:00.944429 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944238 4861 cpu_manager.go:414] "containerMap: removing container" podUID="06ac0898-f2b4-4605-9f7b-2a1dd4628995" containerName="payload-processing" Oct 30 17:14:00.944429 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944246 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ac0898-f2b4-4605-9f7b-2a1dd4628995" containerName="payload-processing" Oct 30 17:14:00.944429 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944293 4861 cpu_manager.go:414] "containerMap: removing container" podUID="af3d6e8a-e6de-4423-b316-0b9ef474905e" containerName="payload-processing" Oct 30 17:14:00.944429 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944301 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af3d6e8a-e6de-4423-b316-0b9ef474905e" containerName="payload-processing" Oct 30 17:14:00.944429 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944335 4861 cpu_manager.go:414] "containerMap: removing container" podUID="c263ff4a-fa28-4eaa-9d93-75a2923f9161" containerName="payload-processing" Oct 30 17:14:00.944429 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944342 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c263ff4a-fa28-4eaa-9d93-75a2923f9161" containerName="payload-processing" Oct 30 17:14:00.944429 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944359 4861 cpu_manager.go:414] "containerMap: removing container" podUID="978ef4f2-ef94-42ed-abcc-01f26e1e5215" containerName="payload-processing" Oct 30 17:14:00.944429 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944367 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="978ef4f2-ef94-42ed-abcc-01f26e1e5215" containerName="payload-processing" Oct 30 17:14:00.944429 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944390 4861 cpu_manager.go:414] "containerMap: removing container" podUID="c2ec4f78-379e-4f7b-9b08-686be0d0524a" containerName="payload-processing" Oct 30 17:14:00.944429 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944398 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2ec4f78-379e-4f7b-9b08-686be0d0524a" containerName="payload-processing" Oct 30 17:14:00.944993 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944588 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ac0898-f2b4-4605-9f7b-2a1dd4628995" containerName="payload-processing" Oct 30 17:14:00.944993 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944611 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="af3d6e8a-e6de-4423-b316-0b9ef474905e" containerName="payload-processing" Oct 30 17:14:00.944993 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944626 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="c263ff4a-fa28-4eaa-9d93-75a2923f9161" containerName="payload-processing" Oct 30 17:14:00.944993 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944637 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="978ef4f2-ef94-42ed-abcc-01f26e1e5215" containerName="payload-processing" Oct 30 17:14:00.944993 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944647 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2ec4f78-379e-4f7b-9b08-686be0d0524a" containerName="payload-processing" Oct 30 17:14:00.944993 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.944657 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4c05e79-f99c-4b1b-8a48-3e366b6f5472" containerName="payload-processing" Oct 30 17:14:00.951394 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.951363 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:00.973221 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.973194 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr"] Oct 30 17:14:00.979373 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:00.979319 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9afbb2-openshift-default-dockercfg-vkv8d\"" Oct 30 17:14:01.061371 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.061329 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce90a319-7468-4c37-8065-01f08642df9d-istiod-ca-cert\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.061536 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.061381 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrjqg\" (UniqueName: \"kubernetes.io/projected/ce90a319-7468-4c37-8065-01f08642df9d-kube-api-access-vrjqg\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.061536 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.061481 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce90a319-7468-4c37-8065-01f08642df9d-istio-podinfo\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.061624 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.061537 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-istio-envoy\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.061664 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.061651 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-istio-data\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.061780 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.061763 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-workload-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.061872 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.061859 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-workload-certs\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.061915 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.061903 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-credential-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.061950 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.061931 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce90a319-7468-4c37-8065-01f08642df9d-istio-token\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.163254 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.163217 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-workload-certs\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.163402 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.163284 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-credential-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.163402 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.163316 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce90a319-7468-4c37-8065-01f08642df9d-istio-token\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.163506 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.163410 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce90a319-7468-4c37-8065-01f08642df9d-istiod-ca-cert\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.163506 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.163454 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrjqg\" (UniqueName: \"kubernetes.io/projected/ce90a319-7468-4c37-8065-01f08642df9d-kube-api-access-vrjqg\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.163594 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.163524 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce90a319-7468-4c37-8065-01f08642df9d-istio-podinfo\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.163634 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.163596 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-istio-envoy\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.163687 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.163655 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-istio-data\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.163806 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.163683 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-workload-certs\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.163806 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.163717 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-credential-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.163922 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.163813 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-workload-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.164030 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.164002 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-istio-data\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.164168 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.164149 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-workload-socket\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.164225 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.164204 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce90a319-7468-4c37-8065-01f08642df9d-istiod-ca-cert\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.166253 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.166227 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-istio-envoy\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.166424 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.166410 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce90a319-7468-4c37-8065-01f08642df9d-istio-podinfo\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.190436 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.190383 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce90a319-7468-4c37-8065-01f08642df9d-istio-token\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.191009 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.190953 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrjqg\" (UniqueName: \"kubernetes.io/projected/ce90a319-7468-4c37-8065-01f08642df9d-kube-api-access-vrjqg\") pod \"e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.281105 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.280823 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:01.733840 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.733809 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr"] Oct 30 17:14:01.741324 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.741285 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:14:01.741570 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.741540 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:14:01.741668 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.741598 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:14:01.811840 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:01.811811 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4c05e79-f99c-4b1b-8a48-3e366b6f5472" path="/var/lib/kubelet/pods/f4c05e79-f99c-4b1b-8a48-3e366b6f5472/volumes" Oct 30 17:14:02.409366 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:02.409127 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" event={"ID":"ce90a319-7468-4c37-8065-01f08642df9d","Type":"ContainerStarted","Data":"de35be27d45cdd31e9edbed0c278c82b2c295ef510c523af0ef06e4ddeb0f1db"} Oct 30 17:14:02.409366 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:02.409289 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" event={"ID":"ce90a319-7468-4c37-8065-01f08642df9d","Type":"ContainerStarted","Data":"ef5f642c5308e373bdf6ecbdeb45dc4aaceb5b4678b6fecd6699aa924ba7bb6f"} Oct 30 17:14:02.491128 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:02.491059 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" podStartSLOduration=2.490972016 podStartE2EDuration="2.490972016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:02.473569419 +0000 UTC m=+3072.816243457" watchObservedRunningTime="2026-10-30 17:14:02.490972016 +0000 UTC m=+3072.833646060" Oct 30 17:14:03.293233 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:03.293199 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:03.347838 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:03.347490 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:03.419427 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:03.419392 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:03.420408 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:03.420370 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:14:04.313982 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.313948 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5"] Oct 30 17:14:04.319792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.319715 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:04.340395 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.340371 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5"] Oct 30 17:14:04.349560 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.349518 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"agc-cred-0eb63fa211fc\"" Oct 30 17:14:04.349712 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.349608 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"openshift-service-ca.crt\"" Oct 30 17:14:04.349712 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.349644 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"kube-root-ca.crt\"" Oct 30 17:14:04.349885 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.349753 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"payload-processing-external-model-e2e--8e3f1132-dockercfg-7xvpm\"" Oct 30 17:14:04.351208 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.351189 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2589b5\"" Oct 30 17:14:04.351336 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.351241 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2589b5\"/\"routing-overlay\"" Oct 30 17:14:04.406300 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.406271 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c12fa8a-3587-4564-b927-c8f036e24997-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5\" (UID: \"8c12fa8a-3587-4564-b927-c8f036e24997\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:04.406463 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.406343 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrcr9\" (UniqueName: \"kubernetes.io/projected/8c12fa8a-3587-4564-b927-c8f036e24997-kube-api-access-hrcr9\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5\" (UID: \"8c12fa8a-3587-4564-b927-c8f036e24997\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:04.406542 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.406484 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c12fa8a-3587-4564-b927-c8f036e24997-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5\" (UID: \"8c12fa8a-3587-4564-b927-c8f036e24997\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:04.406634 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.406613 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c12fa8a-3587-4564-b927-c8f036e24997-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5\" (UID: \"8c12fa8a-3587-4564-b927-c8f036e24997\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:04.509889 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.508688 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c12fa8a-3587-4564-b927-c8f036e24997-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5\" (UID: \"8c12fa8a-3587-4564-b927-c8f036e24997\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:04.509889 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.508963 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c12fa8a-3587-4564-b927-c8f036e24997-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5\" (UID: \"8c12fa8a-3587-4564-b927-c8f036e24997\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:04.509889 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.509140 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrcr9\" (UniqueName: \"kubernetes.io/projected/8c12fa8a-3587-4564-b927-c8f036e24997-kube-api-access-hrcr9\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5\" (UID: \"8c12fa8a-3587-4564-b927-c8f036e24997\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:04.509889 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.509217 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c12fa8a-3587-4564-b927-c8f036e24997-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5\" (UID: \"8c12fa8a-3587-4564-b927-c8f036e24997\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:04.545747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.519947 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c12fa8a-3587-4564-b927-c8f036e24997-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5\" (UID: \"8c12fa8a-3587-4564-b927-c8f036e24997\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:04.545747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.521523 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c12fa8a-3587-4564-b927-c8f036e24997-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5\" (UID: \"8c12fa8a-3587-4564-b927-c8f036e24997\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:04.546774 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.546740 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c12fa8a-3587-4564-b927-c8f036e24997-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5\" (UID: \"8c12fa8a-3587-4564-b927-c8f036e24997\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:04.566551 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.566476 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrcr9\" (UniqueName: \"kubernetes.io/projected/8c12fa8a-3587-4564-b927-c8f036e24997-kube-api-access-hrcr9\") pod \"payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5\" (UID: \"8c12fa8a-3587-4564-b927-c8f036e24997\") " pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:04.641056 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.641020 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:04.973068 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:14:04.973036 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c12fa8a_3587_4564_b927_c8f036e24997.slice/crio-900fab42f2f84c6fab5546c8091f4c16b2463600d8cb8a71f6ebb16cb5ba5aa0 WatchSource:0}: Error finding container 900fab42f2f84c6fab5546c8091f4c16b2463600d8cb8a71f6ebb16cb5ba5aa0: Status 404 returned error can't find the container with id 900fab42f2f84c6fab5546c8091f4c16b2463600d8cb8a71f6ebb16cb5ba5aa0 Oct 30 17:14:04.973327 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:04.973300 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5"] Oct 30 17:14:05.221542 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:05.221444 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz"] Oct 30 17:14:05.228056 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:05.228029 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" Oct 30 17:14:05.246022 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:05.245993 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6e08bd\"" Oct 30 17:14:05.246022 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:05.246007 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz"] Oct 30 17:14:05.327773 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:05.327710 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1995227-e125-48f5-8bf8-ff46ae4fc16e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz\" (UID: \"b1995227-e125-48f5-8bf8-ff46ae4fc16e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" Oct 30 17:14:05.328265 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:05.327839 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jqbl\" (UniqueName: \"kubernetes.io/projected/b1995227-e125-48f5-8bf8-ff46ae4fc16e-kube-api-access-5jqbl\") pod \"payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz\" (UID: \"b1995227-e125-48f5-8bf8-ff46ae4fc16e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" Oct 30 17:14:05.429322 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:05.429291 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1995227-e125-48f5-8bf8-ff46ae4fc16e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz\" (UID: \"b1995227-e125-48f5-8bf8-ff46ae4fc16e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" Oct 30 17:14:05.429484 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:05.429354 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jqbl\" (UniqueName: \"kubernetes.io/projected/b1995227-e125-48f5-8bf8-ff46ae4fc16e-kube-api-access-5jqbl\") pod \"payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz\" (UID: \"b1995227-e125-48f5-8bf8-ff46ae4fc16e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" Oct 30 17:14:05.430242 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:05.430216 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1995227-e125-48f5-8bf8-ff46ae4fc16e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz\" (UID: \"b1995227-e125-48f5-8bf8-ff46ae4fc16e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" Oct 30 17:14:05.438147 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:05.438115 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" event={"ID":"8c12fa8a-3587-4564-b927-c8f036e24997","Type":"ContainerStarted","Data":"6dfde7b2b4bc5fb07414b347b7d38dc9c4f411f20e2f57eb61eddcc7c58da69f"} Oct 30 17:14:05.438301 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:05.438178 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" event={"ID":"8c12fa8a-3587-4564-b927-c8f036e24997","Type":"ContainerStarted","Data":"900fab42f2f84c6fab5546c8091f4c16b2463600d8cb8a71f6ebb16cb5ba5aa0"} Oct 30 17:14:05.438365 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:05.438340 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:05.488023 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:05.487944 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jqbl\" (UniqueName: \"kubernetes.io/projected/b1995227-e125-48f5-8bf8-ff46ae4fc16e-kube-api-access-5jqbl\") pod \"payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz\" (UID: \"b1995227-e125-48f5-8bf8-ff46ae4fc16e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" Oct 30 17:14:05.557056 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:05.557019 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" Oct 30 17:14:06.109255 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:06.109203 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" podStartSLOduration=2.109186696 podStartE2EDuration="2.109186696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:05.5482421 +0000 UTC m=+3075.890916152" watchObservedRunningTime="2026-10-30 17:14:06.109186696 +0000 UTC m=+3076.451860734" Oct 30 17:14:06.110279 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:06.110258 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz"] Oct 30 17:14:06.110491 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:14:06.110466 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1995227_e125_48f5_8bf8_ff46ae4fc16e.slice/crio-61769506548c843650a2101f7a9f93c66b117fc1d2a77b446f0432509c6f93a0 WatchSource:0}: Error finding container 61769506548c843650a2101f7a9f93c66b117fc1d2a77b446f0432509c6f93a0: Status 404 returned error can't find the container with id 61769506548c843650a2101f7a9f93c66b117fc1d2a77b446f0432509c6f93a0 Oct 30 17:14:06.448227 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:06.448123 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" event={"ID":"b1995227-e125-48f5-8bf8-ff46ae4fc16e","Type":"ContainerStarted","Data":"8f7d664e5d4c3424f82afa24419d27a9efea402cd3c4b898ea5080a6a322422d"} Oct 30 17:14:06.448227 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:06.448194 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" event={"ID":"b1995227-e125-48f5-8bf8-ff46ae4fc16e","Type":"ContainerStarted","Data":"61769506548c843650a2101f7a9f93c66b117fc1d2a77b446f0432509c6f93a0"} Oct 30 17:14:06.448819 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:06.448375 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" Oct 30 17:14:06.492705 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:06.492655 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" podStartSLOduration=1.492642152 podStartE2EDuration="1.492642152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:06.492024401 +0000 UTC m=+3076.834698431" watchObservedRunningTime="2026-10-30 17:14:06.492642152 +0000 UTC m=+3076.835316193" Oct 30 17:14:07.038303 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.038268 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l"] Oct 30 17:14:07.043989 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.043966 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:07.064521 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.064498 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l"] Oct 30 17:14:07.073134 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.073102 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"openshift-service-ca.crt\"" Oct 30 17:14:07.073252 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.073133 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5c6b3c\"" Oct 30 17:14:07.073252 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.073138 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"agc-cred-42ac7c0acbd1\"" Oct 30 17:14:07.073252 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.073159 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"routing-overlay\"" Oct 30 17:14:07.073252 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.073209 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"payload-processing-external-model-e2e--22b3ecb4-dockercfg-s8xbn\"" Oct 30 17:14:07.074374 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.074349 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5c6b3c\"/\"kube-root-ca.crt\"" Oct 30 17:14:07.089112 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.089085 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14f4a896-cbf8-4a2c-bbb0-071f00923804-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l\" (UID: \"14f4a896-cbf8-4a2c-bbb0-071f00923804\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:07.089426 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.089404 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdm8g\" (UniqueName: \"kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-kube-api-access-rdm8g\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l\" (UID: \"14f4a896-cbf8-4a2c-bbb0-071f00923804\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:07.089597 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.089578 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l\" (UID: \"14f4a896-cbf8-4a2c-bbb0-071f00923804\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:07.089674 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.089624 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f4a896-cbf8-4a2c-bbb0-071f00923804-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l\" (UID: \"14f4a896-cbf8-4a2c-bbb0-071f00923804\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:07.223749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.216277 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdm8g\" (UniqueName: \"kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-kube-api-access-rdm8g\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l\" (UID: \"14f4a896-cbf8-4a2c-bbb0-071f00923804\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:07.223749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.216570 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l\" (UID: \"14f4a896-cbf8-4a2c-bbb0-071f00923804\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:07.223749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.216642 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f4a896-cbf8-4a2c-bbb0-071f00923804-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l\" (UID: \"14f4a896-cbf8-4a2c-bbb0-071f00923804\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:07.223749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.216689 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14f4a896-cbf8-4a2c-bbb0-071f00923804-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l\" (UID: \"14f4a896-cbf8-4a2c-bbb0-071f00923804\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:07.251744 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.241322 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l\" (UID: \"14f4a896-cbf8-4a2c-bbb0-071f00923804\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:07.251744 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.247869 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f4a896-cbf8-4a2c-bbb0-071f00923804-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l\" (UID: \"14f4a896-cbf8-4a2c-bbb0-071f00923804\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:07.251744 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.248351 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14f4a896-cbf8-4a2c-bbb0-071f00923804-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l\" (UID: \"14f4a896-cbf8-4a2c-bbb0-071f00923804\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:07.265756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.262272 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdm8g\" (UniqueName: \"kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-kube-api-access-rdm8g\") pod \"payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l\" (UID: \"14f4a896-cbf8-4a2c-bbb0-071f00923804\") " pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:07.362801 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.362307 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:07.708386 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:07.708360 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l"] Oct 30 17:14:07.709979 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:14:07.709951 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f4a896_cbf8_4a2c_bbb0_071f00923804.slice/crio-6e4f565aa2a9cbb6df80a5cb3dae985745caeb348dfa6ba55414d3087c6bd802 WatchSource:0}: Error finding container 6e4f565aa2a9cbb6df80a5cb3dae985745caeb348dfa6ba55414d3087c6bd802: Status 404 returned error can't find the container with id 6e4f565aa2a9cbb6df80a5cb3dae985745caeb348dfa6ba55414d3087c6bd802 Oct 30 17:14:08.475673 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:08.475638 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" event={"ID":"14f4a896-cbf8-4a2c-bbb0-071f00923804","Type":"ContainerStarted","Data":"53489fe0f1ae78bd318d75571292f83330929320dd75e4523484670fddc7dca8"} Oct 30 17:14:08.475894 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:08.475699 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" event={"ID":"14f4a896-cbf8-4a2c-bbb0-071f00923804","Type":"ContainerStarted","Data":"6e4f565aa2a9cbb6df80a5cb3dae985745caeb348dfa6ba55414d3087c6bd802"} Oct 30 17:14:08.475962 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:08.475929 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:08.525920 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:08.525873 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" podStartSLOduration=1.525858139 podStartE2EDuration="1.525858139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:08.520705369 +0000 UTC m=+3078.863379408" watchObservedRunningTime="2026-10-30 17:14:08.525858139 +0000 UTC m=+3078.868532177" Oct 30 17:14:09.559834 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:14:09.559796 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-42ac7c0acbd1: secret "agc-cred-42ac7c0acbd1" not found Oct 30 17:14:09.559834 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:14:09.559829 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l: secret "agc-cred-42ac7c0acbd1" not found Oct 30 17:14:09.560272 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:14:09.559905 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-provider-credentials podName:14f4a896-cbf8-4a2c-bbb0-071f00923804 nodeName:}" failed. No retries permitted until 2026-10-30 17:14:10.05988255 +0000 UTC m=+3080.402556571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" (UID: "14f4a896-cbf8-4a2c-bbb0-071f00923804") : secret "agc-cred-42ac7c0acbd1" not found Oct 30 17:14:10.066510 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:14:10.066474 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-42ac7c0acbd1: secret "agc-cred-42ac7c0acbd1" not found Oct 30 17:14:10.066510 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:14:10.066500 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l: secret "agc-cred-42ac7c0acbd1" not found Oct 30 17:14:10.066718 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:14:10.066553 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-provider-credentials podName:14f4a896-cbf8-4a2c-bbb0-071f00923804 nodeName:}" failed. No retries permitted until 2026-10-30 17:14:11.066537703 +0000 UTC m=+3081.409211723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" (UID: "14f4a896-cbf8-4a2c-bbb0-071f00923804") : secret "agc-cred-42ac7c0acbd1" not found Oct 30 17:14:11.007479 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:11.007451 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b"] Oct 30 17:14:11.012578 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:11.012559 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" Oct 30 17:14:11.031483 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:11.031456 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b"] Oct 30 17:14:11.040890 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:11.040864 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9afbb2-dockercfg-7srnp\"" Oct 30 17:14:11.040890 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:11.040882 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9afbb2\"" Oct 30 17:14:11.083626 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:14:11.083593 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-42ac7c0acbd1: secret "agc-cred-42ac7c0acbd1" not found Oct 30 17:14:11.083883 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:14:11.083858 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l: secret "agc-cred-42ac7c0acbd1" not found Oct 30 17:14:11.084024 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:11.083615 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af41a080-c055-4886-8143-d586a24b1164-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b\" (UID: \"af41a080-c055-4886-8143-d586a24b1164\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" Oct 30 17:14:11.084024 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:14:11.083931 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-provider-credentials podName:14f4a896-cbf8-4a2c-bbb0-071f00923804 nodeName:}" failed. No retries permitted until 2026-10-30 17:14:13.083911831 +0000 UTC m=+3083.426585864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" (UID: "14f4a896-cbf8-4a2c-bbb0-071f00923804") : secret "agc-cred-42ac7c0acbd1" not found Oct 30 17:14:11.084169 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:11.084093 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mplhs\" (UniqueName: \"kubernetes.io/projected/af41a080-c055-4886-8143-d586a24b1164-kube-api-access-mplhs\") pod \"payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b\" (UID: \"af41a080-c055-4886-8143-d586a24b1164\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" Oct 30 17:14:11.205747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:11.199167 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mplhs\" (UniqueName: \"kubernetes.io/projected/af41a080-c055-4886-8143-d586a24b1164-kube-api-access-mplhs\") pod \"payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b\" (UID: \"af41a080-c055-4886-8143-d586a24b1164\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" Oct 30 17:14:11.206232 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:11.206196 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af41a080-c055-4886-8143-d586a24b1164-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b\" (UID: \"af41a080-c055-4886-8143-d586a24b1164\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" Oct 30 17:14:11.264803 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:11.251688 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af41a080-c055-4886-8143-d586a24b1164-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b\" (UID: \"af41a080-c055-4886-8143-d586a24b1164\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" Oct 30 17:14:11.276744 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:11.274650 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mplhs\" (UniqueName: \"kubernetes.io/projected/af41a080-c055-4886-8143-d586a24b1164-kube-api-access-mplhs\") pod \"payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b\" (UID: \"af41a080-c055-4886-8143-d586a24b1164\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" Oct 30 17:14:11.331985 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:11.331661 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" Oct 30 17:14:11.435761 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:11.435576 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l"] Oct 30 17:14:11.672899 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:11.672870 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b"] Oct 30 17:14:11.673762 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:14:11.673688 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf41a080_c055_4886_8143_d586a24b1164.slice/crio-6e004c5c52ae191303c5f6e89cc2fe794338af673b58af2315d898c5082677bd WatchSource:0}: Error finding container 6e004c5c52ae191303c5f6e89cc2fe794338af673b58af2315d898c5082677bd: Status 404 returned error can't find the container with id 6e004c5c52ae191303c5f6e89cc2fe794338af673b58af2315d898c5082677bd Oct 30 17:14:12.519635 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:12.519600 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" event={"ID":"af41a080-c055-4886-8143-d586a24b1164","Type":"ContainerStarted","Data":"ced68447fdb3b5770867fa96982bd4c05daea2681675b7e4e9db595072ad9b6b"} Oct 30 17:14:12.520090 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:12.519657 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" event={"ID":"af41a080-c055-4886-8143-d586a24b1164","Type":"ContainerStarted","Data":"6e004c5c52ae191303c5f6e89cc2fe794338af673b58af2315d898c5082677bd"} Oct 30 17:14:12.520090 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:12.519679 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" Oct 30 17:14:12.569969 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:12.569916 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" podStartSLOduration=2.56990207 podStartE2EDuration="2.56990207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:12.563696227 +0000 UTC m=+3082.906370275" watchObservedRunningTime="2026-10-30 17:14:12.56990207 +0000 UTC m=+3082.912576145" Oct 30 17:14:13.097574 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:14:13.097540 4861 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-5c6b3c/agc-cred-42ac7c0acbd1: secret "agc-cred-42ac7c0acbd1" not found Oct 30 17:14:13.097574 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:14:13.097570 4861 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l: secret "agc-cred-42ac7c0acbd1" not found Oct 30 17:14:13.097844 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:14:13.097638 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-provider-credentials podName:14f4a896-cbf8-4a2c-bbb0-071f00923804 nodeName:}" failed. No retries permitted until 2026-10-30 17:14:17.097618134 +0000 UTC m=+3087.440292163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" (UID: "14f4a896-cbf8-4a2c-bbb0-071f00923804") : secret "agc-cred-42ac7c0acbd1" not found Oct 30 17:14:13.204688 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:13.204652 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5"] Oct 30 17:14:13.208418 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:13.208390 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:16.437151 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:16.437101 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" podUID="14f4a896-cbf8-4a2c-bbb0-071f00923804" containerName="payload-processing" containerID="cri-o://53489fe0f1ae78bd318d75571292f83330929320dd75e4523484670fddc7dca8" gracePeriod=25 Oct 30 17:14:16.623769 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:16.623736 4861 generic.go:356] "Generic (PLEG): container finished" podID="14f4a896-cbf8-4a2c-bbb0-071f00923804" containerID="53489fe0f1ae78bd318d75571292f83330929320dd75e4523484670fddc7dca8" exitCode=0 Oct 30 17:14:16.623932 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:16.623796 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" event={"ID":"14f4a896-cbf8-4a2c-bbb0-071f00923804","Type":"ContainerDied","Data":"53489fe0f1ae78bd318d75571292f83330929320dd75e4523484670fddc7dca8"} Oct 30 17:14:16.764419 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:16.764393 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:16.770464 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:16.770437 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14f4a896-cbf8-4a2c-bbb0-071f00923804-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14f4a896-cbf8-4a2c-bbb0-071f00923804-routing-overlay-volume\") pod \"14f4a896-cbf8-4a2c-bbb0-071f00923804\" (UID: \"14f4a896-cbf8-4a2c-bbb0-071f00923804\") " Oct 30 17:14:16.770599 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:16.770486 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-kube-api-access-rdm8g\" (UniqueName: \"kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-kube-api-access-rdm8g\") pod \"14f4a896-cbf8-4a2c-bbb0-071f00923804\" (UID: \"14f4a896-cbf8-4a2c-bbb0-071f00923804\") " Oct 30 17:14:16.770667 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:16.770631 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-provider-credentials\" (UniqueName: \"kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-provider-credentials\") pod \"14f4a896-cbf8-4a2c-bbb0-071f00923804\" (UID: \"14f4a896-cbf8-4a2c-bbb0-071f00923804\") " Oct 30 17:14:16.770794 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:16.770772 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f4a896-cbf8-4a2c-bbb0-071f00923804-routing-overlay-volume" pod "14f4a896-cbf8-4a2c-bbb0-071f00923804" (UID: "14f4a896-cbf8-4a2c-bbb0-071f00923804"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:16.770879 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:16.770822 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14f4a896-cbf8-4a2c-bbb0-071f00923804-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f4a896-cbf8-4a2c-bbb0-071f00923804-plugins-config-volume\") pod \"14f4a896-cbf8-4a2c-bbb0-071f00923804\" (UID: \"14f4a896-cbf8-4a2c-bbb0-071f00923804\") " Oct 30 17:14:16.771202 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:16.771176 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14f4a896-cbf8-4a2c-bbb0-071f00923804-plugins-config-volume" pod "14f4a896-cbf8-4a2c-bbb0-071f00923804" (UID: "14f4a896-cbf8-4a2c-bbb0-071f00923804"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:16.771450 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:16.771353 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14f4a896-cbf8-4a2c-bbb0-071f00923804-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:14:16.771450 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:16.771370 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/14f4a896-cbf8-4a2c-bbb0-071f00923804-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:14:16.772901 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:16.772882 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-kube-api-access-rdm8g" pod "14f4a896-cbf8-4a2c-bbb0-071f00923804" (UID: "14f4a896-cbf8-4a2c-bbb0-071f00923804"). InnerVolumeSpecName "kube-api-access-rdm8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:16.772998 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:16.772927 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-provider-credentials" pod "14f4a896-cbf8-4a2c-bbb0-071f00923804" (UID: "14f4a896-cbf8-4a2c-bbb0-071f00923804"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:16.872912 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:16.872884 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:14:16.872912 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:16.872909 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdm8g\" (UniqueName: \"kubernetes.io/projected/14f4a896-cbf8-4a2c-bbb0-071f00923804-kube-api-access-rdm8g\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:14:17.463918 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:17.463885 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" Oct 30 17:14:17.742519 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:17.735540 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" Oct 30 17:14:17.743915 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:17.743617 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l" event={"ID":"14f4a896-cbf8-4a2c-bbb0-071f00923804","Type":"ContainerDied","Data":"6e4f565aa2a9cbb6df80a5cb3dae985745caeb348dfa6ba55414d3087c6bd802"} Oct 30 17:14:17.743915 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:17.743717 4861 scope.go:122] "RemoveContainer" containerID="53489fe0f1ae78bd318d75571292f83330929320dd75e4523484670fddc7dca8" Oct 30 17:14:17.829848 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:17.829809 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l"] Oct 30 17:14:17.830069 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:17.829866 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5c6b3c/payload-processing-external-model-e2e-extmod-b-5c6b3c-5fd8zrr6l"] Oct 30 17:14:18.205539 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.205445 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" podUID="8c12fa8a-3587-4564-b927-c8f036e24997" containerName="payload-processing" containerID="cri-o://6dfde7b2b4bc5fb07414b347b7d38dc9c4f411f20e2f57eb61eddcc7c58da69f" gracePeriod=25 Oct 30 17:14:18.488815 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.488784 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:18.561945 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.561902 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c12fa8a-3587-4564-b927-c8f036e24997-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c12fa8a-3587-4564-b927-c8f036e24997-routing-overlay-volume\") pod \"8c12fa8a-3587-4564-b927-c8f036e24997\" (UID: \"8c12fa8a-3587-4564-b927-c8f036e24997\") " Oct 30 17:14:18.562108 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.561964 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c12fa8a-3587-4564-b927-c8f036e24997-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c12fa8a-3587-4564-b927-c8f036e24997-plugins-config-volume\") pod \"8c12fa8a-3587-4564-b927-c8f036e24997\" (UID: \"8c12fa8a-3587-4564-b927-c8f036e24997\") " Oct 30 17:14:18.562212 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.562116 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c12fa8a-3587-4564-b927-c8f036e24997-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c12fa8a-3587-4564-b927-c8f036e24997-provider-credentials\") pod \"8c12fa8a-3587-4564-b927-c8f036e24997\" (UID: \"8c12fa8a-3587-4564-b927-c8f036e24997\") " Oct 30 17:14:18.562429 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.562406 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c12fa8a-3587-4564-b927-c8f036e24997-routing-overlay-volume" pod "8c12fa8a-3587-4564-b927-c8f036e24997" (UID: "8c12fa8a-3587-4564-b927-c8f036e24997"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:18.562429 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.562419 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c12fa8a-3587-4564-b927-c8f036e24997-kube-api-access-hrcr9\" (UniqueName: \"kubernetes.io/projected/8c12fa8a-3587-4564-b927-c8f036e24997-kube-api-access-hrcr9\") pod \"8c12fa8a-3587-4564-b927-c8f036e24997\" (UID: \"8c12fa8a-3587-4564-b927-c8f036e24997\") " Oct 30 17:14:18.562613 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.562437 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c12fa8a-3587-4564-b927-c8f036e24997-plugins-config-volume" pod "8c12fa8a-3587-4564-b927-c8f036e24997" (UID: "8c12fa8a-3587-4564-b927-c8f036e24997"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:18.563140 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.563116 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8c12fa8a-3587-4564-b927-c8f036e24997-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.563272 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.563143 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c12fa8a-3587-4564-b927-c8f036e24997-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.564901 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.564868 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c12fa8a-3587-4564-b927-c8f036e24997-provider-credentials" pod "8c12fa8a-3587-4564-b927-c8f036e24997" (UID: "8c12fa8a-3587-4564-b927-c8f036e24997"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:18.565311 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.565293 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c12fa8a-3587-4564-b927-c8f036e24997-kube-api-access-hrcr9" pod "8c12fa8a-3587-4564-b927-c8f036e24997" (UID: "8c12fa8a-3587-4564-b927-c8f036e24997"). InnerVolumeSpecName "kube-api-access-hrcr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:18.664326 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.664299 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrcr9\" (UniqueName: \"kubernetes.io/projected/8c12fa8a-3587-4564-b927-c8f036e24997-kube-api-access-hrcr9\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.664326 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.664328 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8c12fa8a-3587-4564-b927-c8f036e24997-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.754453 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.754424 4861 generic.go:356] "Generic (PLEG): container finished" podID="8c12fa8a-3587-4564-b927-c8f036e24997" containerID="6dfde7b2b4bc5fb07414b347b7d38dc9c4f411f20e2f57eb61eddcc7c58da69f" exitCode=0 Oct 30 17:14:18.754607 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.754473 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" Oct 30 17:14:18.754607 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.754510 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" event={"ID":"8c12fa8a-3587-4564-b927-c8f036e24997","Type":"ContainerDied","Data":"6dfde7b2b4bc5fb07414b347b7d38dc9c4f411f20e2f57eb61eddcc7c58da69f"} Oct 30 17:14:18.754607 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.754561 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5" event={"ID":"8c12fa8a-3587-4564-b927-c8f036e24997","Type":"ContainerDied","Data":"900fab42f2f84c6fab5546c8091f4c16b2463600d8cb8a71f6ebb16cb5ba5aa0"} Oct 30 17:14:18.754607 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.754582 4861 scope.go:122] "RemoveContainer" containerID="6dfde7b2b4bc5fb07414b347b7d38dc9c4f411f20e2f57eb61eddcc7c58da69f" Oct 30 17:14:18.767265 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.767099 4861 scope.go:122] "RemoveContainer" containerID="6dfde7b2b4bc5fb07414b347b7d38dc9c4f411f20e2f57eb61eddcc7c58da69f" Oct 30 17:14:18.767396 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:14:18.767376 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dfde7b2b4bc5fb07414b347b7d38dc9c4f411f20e2f57eb61eddcc7c58da69f\": container with ID starting with 6dfde7b2b4bc5fb07414b347b7d38dc9c4f411f20e2f57eb61eddcc7c58da69f not found: ID does not exist" containerID="6dfde7b2b4bc5fb07414b347b7d38dc9c4f411f20e2f57eb61eddcc7c58da69f" Oct 30 17:14:18.767466 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.767409 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dfde7b2b4bc5fb07414b347b7d38dc9c4f411f20e2f57eb61eddcc7c58da69f"} err="failed to get container status \"6dfde7b2b4bc5fb07414b347b7d38dc9c4f411f20e2f57eb61eddcc7c58da69f\": rpc error: code = NotFound desc = could not find container \"6dfde7b2b4bc5fb07414b347b7d38dc9c4f411f20e2f57eb61eddcc7c58da69f\": container with ID starting with 6dfde7b2b4bc5fb07414b347b7d38dc9c4f411f20e2f57eb61eddcc7c58da69f not found: ID does not exist" Oct 30 17:14:18.813888 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.813866 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5"] Oct 30 17:14:18.827564 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:18.827541 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2589b5/payload-processing-external-model-e2e-extmod-a-2589b5-6945rxdz5"] Oct 30 17:14:19.894390 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:19.894355 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14f4a896-cbf8-4a2c-bbb0-071f00923804" path="/var/lib/kubelet/pods/14f4a896-cbf8-4a2c-bbb0-071f00923804/volumes" Oct 30 17:14:19.895116 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:19.895094 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c12fa8a-3587-4564-b927-c8f036e24997" path="/var/lib/kubelet/pods/8c12fa8a-3587-4564-b927-c8f036e24997/volumes" Oct 30 17:14:20.292121 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:14:20.292090 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:14:23.418631 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.417683 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc"] Oct 30 17:14:23.419756 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.419713 4861 cpu_manager.go:414] "containerMap: removing container" podUID="14f4a896-cbf8-4a2c-bbb0-071f00923804" containerName="payload-processing" Oct 30 17:14:23.419965 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.419940 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14f4a896-cbf8-4a2c-bbb0-071f00923804" containerName="payload-processing" Oct 30 17:14:23.420099 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.420087 4861 cpu_manager.go:414] "containerMap: removing container" podUID="8c12fa8a-3587-4564-b927-c8f036e24997" containerName="payload-processing" Oct 30 17:14:23.420155 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.420099 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c12fa8a-3587-4564-b927-c8f036e24997" containerName="payload-processing" Oct 30 17:14:23.420546 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.420524 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c12fa8a-3587-4564-b927-c8f036e24997" containerName="payload-processing" Oct 30 17:14:23.420653 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.420568 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="14f4a896-cbf8-4a2c-bbb0-071f00923804" containerName="payload-processing" Oct 30 17:14:23.428118 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.428097 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" Oct 30 17:14:23.445088 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.445064 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc"] Oct 30 17:14:23.452378 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.452358 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e15d1f-dockercfg-rjm4g\"" Oct 30 17:14:23.452500 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.452364 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e15d1f\"" Oct 30 17:14:23.503511 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.503484 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt"] Oct 30 17:14:23.509090 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.509066 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" Oct 30 17:14:23.530180 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.530139 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt"] Oct 30 17:14:23.574778 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.568684 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6013ebf2-30f0-42a9-a31f-e66795df2b56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc\" (UID: \"6013ebf2-30f0-42a9-a31f-e66795df2b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" Oct 30 17:14:23.740480 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.736493 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzsqt\" (UniqueName: \"kubernetes.io/projected/6013ebf2-30f0-42a9-a31f-e66795df2b56-kube-api-access-tzsqt\") pod \"payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc\" (UID: \"6013ebf2-30f0-42a9-a31f-e66795df2b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" Oct 30 17:14:23.741031 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.735936 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" Oct 30 17:14:23.843492 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.843451 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6013ebf2-30f0-42a9-a31f-e66795df2b56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc\" (UID: \"6013ebf2-30f0-42a9-a31f-e66795df2b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" Oct 30 17:14:23.843673 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.843537 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce50453-ccab-4825-8ca9-b15579c1d0b4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt\" (UID: \"1ce50453-ccab-4825-8ca9-b15579c1d0b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" Oct 30 17:14:23.844664 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.844628 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6013ebf2-30f0-42a9-a31f-e66795df2b56-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc\" (UID: \"6013ebf2-30f0-42a9-a31f-e66795df2b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" Oct 30 17:14:23.845410 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.845380 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-776c2\" (UniqueName: \"kubernetes.io/projected/1ce50453-ccab-4825-8ca9-b15579c1d0b4-kube-api-access-776c2\") pod \"payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt\" (UID: \"1ce50453-ccab-4825-8ca9-b15579c1d0b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" Oct 30 17:14:23.845537 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.845521 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzsqt\" (UniqueName: \"kubernetes.io/projected/6013ebf2-30f0-42a9-a31f-e66795df2b56-kube-api-access-tzsqt\") pod \"payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc\" (UID: \"6013ebf2-30f0-42a9-a31f-e66795df2b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" Oct 30 17:14:23.873242 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.873213 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzsqt\" (UniqueName: \"kubernetes.io/projected/6013ebf2-30f0-42a9-a31f-e66795df2b56-kube-api-access-tzsqt\") pod \"payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc\" (UID: \"6013ebf2-30f0-42a9-a31f-e66795df2b56\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" Oct 30 17:14:23.947551 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.947458 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce50453-ccab-4825-8ca9-b15579c1d0b4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt\" (UID: \"1ce50453-ccab-4825-8ca9-b15579c1d0b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" Oct 30 17:14:23.947767 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.947571 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-776c2\" (UniqueName: \"kubernetes.io/projected/1ce50453-ccab-4825-8ca9-b15579c1d0b4-kube-api-access-776c2\") pod \"payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt\" (UID: \"1ce50453-ccab-4825-8ca9-b15579c1d0b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" Oct 30 17:14:23.948382 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.948349 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce50453-ccab-4825-8ca9-b15579c1d0b4-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt\" (UID: \"1ce50453-ccab-4825-8ca9-b15579c1d0b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" Oct 30 17:14:23.972992 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:23.972968 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-776c2\" (UniqueName: \"kubernetes.io/projected/1ce50453-ccab-4825-8ca9-b15579c1d0b4-kube-api-access-776c2\") pod \"payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt\" (UID: \"1ce50453-ccab-4825-8ca9-b15579c1d0b4\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" Oct 30 17:14:24.043528 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:24.043463 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" Oct 30 17:14:24.125081 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:24.125053 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" Oct 30 17:14:24.358337 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:14:24.358300 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6013ebf2_30f0_42a9_a31f_e66795df2b56.slice/crio-d6fd2beacaf67907a50c30c88cb353941f56dd4276a5839b35344240583411e0 WatchSource:0}: Error finding container d6fd2beacaf67907a50c30c88cb353941f56dd4276a5839b35344240583411e0: Status 404 returned error can't find the container with id d6fd2beacaf67907a50c30c88cb353941f56dd4276a5839b35344240583411e0 Oct 30 17:14:24.358758 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:24.358716 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc"] Oct 30 17:14:24.465488 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:14:24.465458 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce50453_ccab_4825_8ca9_b15579c1d0b4.slice/crio-7d5a8b0a8a11e379418ef3961eb3c4185ae6e978b7e5ce5a211c08c333c8ae85 WatchSource:0}: Error finding container 7d5a8b0a8a11e379418ef3961eb3c4185ae6e978b7e5ce5a211c08c333c8ae85: Status 404 returned error can't find the container with id 7d5a8b0a8a11e379418ef3961eb3c4185ae6e978b7e5ce5a211c08c333c8ae85 Oct 30 17:14:24.466310 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:24.466275 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt"] Oct 30 17:14:24.821201 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:24.821164 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" event={"ID":"1ce50453-ccab-4825-8ca9-b15579c1d0b4","Type":"ContainerStarted","Data":"2c26e348f0500d26b63327cc84a24fa6ac741c54f142196b671bc37317793858"} Oct 30 17:14:24.821402 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:24.821230 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" Oct 30 17:14:24.821402 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:24.821252 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" event={"ID":"1ce50453-ccab-4825-8ca9-b15579c1d0b4","Type":"ContainerStarted","Data":"7d5a8b0a8a11e379418ef3961eb3c4185ae6e978b7e5ce5a211c08c333c8ae85"} Oct 30 17:14:24.823155 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:24.823122 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" event={"ID":"6013ebf2-30f0-42a9-a31f-e66795df2b56","Type":"ContainerStarted","Data":"0d434a8083452c5d5d095456075efecc78d7926662d0d95d205821c8ba01b6b5"} Oct 30 17:14:24.823839 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:24.823177 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" event={"ID":"6013ebf2-30f0-42a9-a31f-e66795df2b56","Type":"ContainerStarted","Data":"d6fd2beacaf67907a50c30c88cb353941f56dd4276a5839b35344240583411e0"} Oct 30 17:14:24.823839 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:24.823204 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" Oct 30 17:14:24.868089 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:24.868029 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" podStartSLOduration=1.868011413 podStartE2EDuration="1.868011413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:24.864545879 +0000 UTC m=+3095.207219918" watchObservedRunningTime="2026-10-30 17:14:24.868011413 +0000 UTC m=+3095.210685452" Oct 30 17:14:24.912414 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:24.912362 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" podStartSLOduration=1.912348978 podStartE2EDuration="1.912348978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:24.908815872 +0000 UTC m=+3095.251489911" watchObservedRunningTime="2026-10-30 17:14:24.912348978 +0000 UTC m=+3095.255023016" Oct 30 17:14:34.922087 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:34.922054 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c"] Oct 30 17:14:34.927492 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:34.927470 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:34.945115 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:34.945090 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6acc78\"/\"openshift-service-ca.crt\"" Oct 30 17:14:34.945264 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:34.945233 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6acc78\"/\"kube-root-ca.crt\"" Oct 30 17:14:34.949738 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:34.949698 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c"] Oct 30 17:14:34.954692 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:34.954668 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6acc78\"/\"default-dockercfg-dsmcv\"" Oct 30 17:14:34.954877 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:34.954796 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6acc78\"/\"test-model-6acc78-kserve-self-signed-certs\"" Oct 30 17:14:35.045543 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.045517 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-home\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.045677 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.045552 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-dshm\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.045677 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.045646 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-model-cache\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.045773 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.045740 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whzv9\" (UniqueName: \"kubernetes.io/projected/e73e1c71-d9da-4c54-94e7-9862e81a0be5-kube-api-access-whzv9\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.045977 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.045960 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73e1c71-d9da-4c54-94e7-9862e81a0be5-tls-certs\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.046025 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.046013 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-tmp-dir\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.157387 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.156866 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73e1c71-d9da-4c54-94e7-9862e81a0be5-tls-certs\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.157711 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.157689 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-tmp-dir\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.169513 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.169475 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-home\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.169777 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.169753 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-dshm\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.169970 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.169952 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-model-cache\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.170184 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.170166 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whzv9\" (UniqueName: \"kubernetes.io/projected/e73e1c71-d9da-4c54-94e7-9862e81a0be5-kube-api-access-whzv9\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.171058 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.158380 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-tmp-dir\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.174536 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.174478 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-model-cache\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.174867 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.174833 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-home\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.197554 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.197490 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73e1c71-d9da-4c54-94e7-9862e81a0be5-tls-certs\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.217744 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.215513 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whzv9\" (UniqueName: \"kubernetes.io/projected/e73e1c71-d9da-4c54-94e7-9862e81a0be5-kube-api-access-whzv9\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.235134 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.235053 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-dshm\") pod \"test-model-6acc78-kserve-7c5d8c7d9f-pww5c\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.252409 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.252048 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:35.593912 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.593884 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c"] Oct 30 17:14:35.594248 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:14:35.594213 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73e1c71_d9da_4c54_94e7_9862e81a0be5.slice/crio-8ffc138d3ea1637da58163986a93dd29e2b351131affe9e880834b1eb7b00654 WatchSource:0}: Error finding container 8ffc138d3ea1637da58163986a93dd29e2b351131affe9e880834b1eb7b00654: Status 404 returned error can't find the container with id 8ffc138d3ea1637da58163986a93dd29e2b351131affe9e880834b1eb7b00654 Oct 30 17:14:35.836069 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.836044 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" Oct 30 17:14:35.836248 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.836102 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" Oct 30 17:14:35.924807 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.922187 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" event={"ID":"e73e1c71-d9da-4c54-94e7-9862e81a0be5","Type":"ContainerStarted","Data":"c8e4998cf936752cab1ae1f7680ab05d920d5f70ed85600200c0f465ce2b212f"} Oct 30 17:14:35.924807 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.922245 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" event={"ID":"e73e1c71-d9da-4c54-94e7-9862e81a0be5","Type":"ContainerStarted","Data":"8ffc138d3ea1637da58163986a93dd29e2b351131affe9e880834b1eb7b00654"} Oct 30 17:14:35.973712 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:35.973651 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" podStartSLOduration=1.9736380119999999 podStartE2EDuration="1.973638012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:35.970154216 +0000 UTC m=+3106.312828253" watchObservedRunningTime="2026-10-30 17:14:35.973638012 +0000 UTC m=+3106.316312050" Oct 30 17:14:36.253552 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:36.253524 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:45.253650 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:45.253613 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:45.267091 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:45.267067 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:46.018629 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:46.018596 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:14:50.291438 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:14:50.291393 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:14:51.046063 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.046030 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h"] Oct 30 17:14:51.051158 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.051137 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:14:51.099206 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.096381 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6e08bd-serving-cert\"" Oct 30 17:14:51.103506 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.103476 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h"] Oct 30 17:14:51.142510 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.142472 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbr5g\" (UniqueName: \"kubernetes.io/projected/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-kube-api-access-mbr5g\") pod \"maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h\" (UID: \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:14:51.142666 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.142514 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h\" (UID: \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:14:51.142757 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.142665 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-maas-api-tls\") pod \"maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h\" (UID: \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:14:51.142885 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.142828 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-metrics-certs\") pod \"maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h\" (UID: \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:14:51.244196 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.244167 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbr5g\" (UniqueName: \"kubernetes.io/projected/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-kube-api-access-mbr5g\") pod \"maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h\" (UID: \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:14:51.244365 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.244203 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h\" (UID: \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:14:51.244365 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.244254 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-maas-api-tls\") pod \"maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h\" (UID: \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:14:51.244365 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.244308 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-metrics-certs\") pod \"maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h\" (UID: \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:14:51.245096 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.245034 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h\" (UID: \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:14:51.246976 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.246945 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-maas-api-tls\") pod \"maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h\" (UID: \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:14:51.247089 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.247055 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-metrics-certs\") pod \"maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h\" (UID: \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:14:51.269829 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.269809 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbr5g\" (UniqueName: \"kubernetes.io/projected/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-kube-api-access-mbr5g\") pod \"maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h\" (UID: \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:14:51.370805 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.370471 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:14:51.742920 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:51.742889 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h"] Oct 30 17:14:51.743063 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:14:51.743017 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd252b794_0a34_4e07_94ee_2b1c7a2e40d0.slice/crio-b0061032a00f15396db094e7c101d007efb97250e8808becf3f599ec786a4c4b WatchSource:0}: Error finding container b0061032a00f15396db094e7c101d007efb97250e8808becf3f599ec786a4c4b: Status 404 returned error can't find the container with id b0061032a00f15396db094e7c101d007efb97250e8808becf3f599ec786a4c4b Oct 30 17:14:52.058196 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:52.058117 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" event={"ID":"d252b794-0a34-4e07-94ee-2b1c7a2e40d0","Type":"ContainerStarted","Data":"b0061032a00f15396db094e7c101d007efb97250e8808becf3f599ec786a4c4b"} Oct 30 17:14:53.067815 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:53.067777 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" event={"ID":"d252b794-0a34-4e07-94ee-2b1c7a2e40d0","Type":"ContainerStarted","Data":"7896dc664db90fde91f936fa9f139cac7f50043e4a805dc253d7fb7c80cdfdb9"} Oct 30 17:14:53.068267 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:53.067871 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:14:53.117362 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:53.117302 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" podStartSLOduration=1.5322561879999999 podStartE2EDuration="2.117285541s" totalImagesPullingTime="585.029353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:53.114180872 +0000 UTC m=+3123.456854912" watchObservedRunningTime="2026-10-30 17:14:53.117285541 +0000 UTC m=+3123.459959581" Oct 30 17:14:54.945474 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:54.945437 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64"] Oct 30 17:14:54.950716 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:54.950691 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:14:54.973683 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:54.973659 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64"] Oct 30 17:14:54.982069 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:54.982043 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9afbb2-serving-cert\"" Oct 30 17:14:55.041972 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:55.041925 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e40b662-7953-4e26-921b-afb268c58d8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64\" (UID: \"6e40b662-7953-4e26-921b-afb268c58d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:14:55.042199 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:55.042171 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e40b662-7953-4e26-921b-afb268c58d8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64\" (UID: \"6e40b662-7953-4e26-921b-afb268c58d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:14:55.042352 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:55.042331 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctgw6\" (UniqueName: \"kubernetes.io/projected/6e40b662-7953-4e26-921b-afb268c58d8e-kube-api-access-ctgw6\") pod \"maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64\" (UID: \"6e40b662-7953-4e26-921b-afb268c58d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:14:55.042501 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:55.042484 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e40b662-7953-4e26-921b-afb268c58d8e-metrics-certs\") pod \"maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64\" (UID: \"6e40b662-7953-4e26-921b-afb268c58d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:14:55.143985 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:55.143943 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e40b662-7953-4e26-921b-afb268c58d8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64\" (UID: \"6e40b662-7953-4e26-921b-afb268c58d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:14:55.144182 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:55.144028 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctgw6\" (UniqueName: \"kubernetes.io/projected/6e40b662-7953-4e26-921b-afb268c58d8e-kube-api-access-ctgw6\") pod \"maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64\" (UID: \"6e40b662-7953-4e26-921b-afb268c58d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:14:55.144182 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:55.144099 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e40b662-7953-4e26-921b-afb268c58d8e-metrics-certs\") pod \"maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64\" (UID: \"6e40b662-7953-4e26-921b-afb268c58d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:14:55.144328 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:55.144189 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e40b662-7953-4e26-921b-afb268c58d8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64\" (UID: \"6e40b662-7953-4e26-921b-afb268c58d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:14:55.145000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:55.144919 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e40b662-7953-4e26-921b-afb268c58d8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64\" (UID: \"6e40b662-7953-4e26-921b-afb268c58d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:14:55.146692 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:55.146662 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e40b662-7953-4e26-921b-afb268c58d8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64\" (UID: \"6e40b662-7953-4e26-921b-afb268c58d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:14:55.146897 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:55.146878 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e40b662-7953-4e26-921b-afb268c58d8e-metrics-certs\") pod \"maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64\" (UID: \"6e40b662-7953-4e26-921b-afb268c58d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:14:55.169065 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:55.169039 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctgw6\" (UniqueName: \"kubernetes.io/projected/6e40b662-7953-4e26-921b-afb268c58d8e-kube-api-access-ctgw6\") pod \"maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64\" (UID: \"6e40b662-7953-4e26-921b-afb268c58d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:14:55.273474 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:55.272002 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:14:55.613575 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:14:55.613544 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e40b662_7953_4e26_921b_afb268c58d8e.slice/crio-27b1a175c5c38bccbe6e32ef2f245ca015f25bda2f423cef86896c714a949c38 WatchSource:0}: Error finding container 27b1a175c5c38bccbe6e32ef2f245ca015f25bda2f423cef86896c714a949c38: Status 404 returned error can't find the container with id 27b1a175c5c38bccbe6e32ef2f245ca015f25bda2f423cef86896c714a949c38 Oct 30 17:14:55.613927 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:55.613906 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64"] Oct 30 17:14:56.096041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:56.096005 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" event={"ID":"6e40b662-7953-4e26-921b-afb268c58d8e","Type":"ContainerStarted","Data":"4ad2a86de5d5cd10e7c04c133dd006f4e09c5e245127554c12c3924591857845"} Oct 30 17:14:56.096490 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:56.096078 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:14:56.096490 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:56.096105 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" event={"ID":"6e40b662-7953-4e26-921b-afb268c58d8e","Type":"ContainerStarted","Data":"27b1a175c5c38bccbe6e32ef2f245ca015f25bda2f423cef86896c714a949c38"} Oct 30 17:14:56.201751 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:56.183218 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" podStartSLOduration=1.825456701 podStartE2EDuration="2.18319882s" totalImagesPullingTime="357.742119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:14:56.17862569 +0000 UTC m=+3126.521299731" watchObservedRunningTime="2026-10-30 17:14:56.18319882 +0000 UTC m=+3126.525872859" Oct 30 17:14:59.081132 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:14:59.081098 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:15:00.226078 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.226036 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r"] Oct 30 17:15:00.235985 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.235957 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" Oct 30 17:15:00.248223 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.248195 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks"] Oct 30 17:15:00.257364 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.257342 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" Oct 30 17:15:00.257904 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.257885 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r"] Oct 30 17:15:00.264807 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.264778 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b"] Oct 30 17:15:00.275838 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.275814 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" Oct 30 17:15:00.276816 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.276769 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kr7cj\"" Oct 30 17:15:00.278247 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.278209 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks"] Oct 30 17:15:00.285984 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.285951 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp"] Oct 30 17:15:00.293222 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.293203 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" Oct 30 17:15:00.302633 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.302610 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b"] Oct 30 17:15:00.331745 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.331705 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp"] Oct 30 17:15:00.354912 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.354880 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77f6b18-984e-450e-a62b-92795158af88-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889675-tt8ks\" (UID: \"b77f6b18-984e-450e-a62b-92795158af88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" Oct 30 17:15:00.355084 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.354919 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flm8g\" (UniqueName: \"kubernetes.io/projected/b77f6b18-984e-450e-a62b-92795158af88-kube-api-access-flm8g\") pod \"maas-api-key-cleanup-29889675-tt8ks\" (UID: \"b77f6b18-984e-450e-a62b-92795158af88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" Oct 30 17:15:00.355180 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.355095 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhgld\" (UniqueName: \"kubernetes.io/projected/7cafb0bf-0a01-4b90-a658-9966e1a6d155-kube-api-access-hhgld\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b\" (UID: \"7cafb0bf-0a01-4b90-a658-9966e1a6d155\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" Oct 30 17:15:00.355275 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.355254 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/880daa36-57b6-4055-ab40-224cbbbe9f5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r\" (UID: \"880daa36-57b6-4055-ab40-224cbbbe9f5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" Oct 30 17:15:00.355345 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.355320 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlrxb\" (UniqueName: \"kubernetes.io/projected/880daa36-57b6-4055-ab40-224cbbbe9f5d-kube-api-access-wlrxb\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r\" (UID: \"880daa36-57b6-4055-ab40-224cbbbe9f5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" Oct 30 17:15:00.355445 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.355418 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cafb0bf-0a01-4b90-a658-9966e1a6d155-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b\" (UID: \"7cafb0bf-0a01-4b90-a658-9966e1a6d155\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" Oct 30 17:15:00.457210 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.457175 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/880daa36-57b6-4055-ab40-224cbbbe9f5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r\" (UID: \"880daa36-57b6-4055-ab40-224cbbbe9f5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" Oct 30 17:15:00.457408 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.457236 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrxb\" (UniqueName: \"kubernetes.io/projected/880daa36-57b6-4055-ab40-224cbbbe9f5d-kube-api-access-wlrxb\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r\" (UID: \"880daa36-57b6-4055-ab40-224cbbbe9f5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" Oct 30 17:15:00.457408 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.457269 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cafb0bf-0a01-4b90-a658-9966e1a6d155-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b\" (UID: \"7cafb0bf-0a01-4b90-a658-9966e1a6d155\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" Oct 30 17:15:00.457500 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.457480 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77f6b18-984e-450e-a62b-92795158af88-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889675-tt8ks\" (UID: \"b77f6b18-984e-450e-a62b-92795158af88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" Oct 30 17:15:00.457543 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.457514 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flm8g\" (UniqueName: \"kubernetes.io/projected/b77f6b18-984e-450e-a62b-92795158af88-kube-api-access-flm8g\") pod \"maas-api-key-cleanup-29889675-tt8ks\" (UID: \"b77f6b18-984e-450e-a62b-92795158af88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" Oct 30 17:15:00.457603 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.457586 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbpts\" (UniqueName: \"kubernetes.io/projected/945a95c0-1afa-41a8-b23f-80db41dd13a7-kube-api-access-dbpts\") pod \"maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp\" (UID: \"945a95c0-1afa-41a8-b23f-80db41dd13a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" Oct 30 17:15:00.457689 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.457651 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/945a95c0-1afa-41a8-b23f-80db41dd13a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp\" (UID: \"945a95c0-1afa-41a8-b23f-80db41dd13a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" Oct 30 17:15:00.457773 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.457759 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhgld\" (UniqueName: \"kubernetes.io/projected/7cafb0bf-0a01-4b90-a658-9966e1a6d155-kube-api-access-hhgld\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b\" (UID: \"7cafb0bf-0a01-4b90-a658-9966e1a6d155\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" Oct 30 17:15:00.458279 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.458242 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/880daa36-57b6-4055-ab40-224cbbbe9f5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r\" (UID: \"880daa36-57b6-4055-ab40-224cbbbe9f5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" Oct 30 17:15:00.458416 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.458346 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cafb0bf-0a01-4b90-a658-9966e1a6d155-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b\" (UID: \"7cafb0bf-0a01-4b90-a658-9966e1a6d155\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" Oct 30 17:15:00.458416 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.458363 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77f6b18-984e-450e-a62b-92795158af88-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889675-tt8ks\" (UID: \"b77f6b18-984e-450e-a62b-92795158af88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" Oct 30 17:15:00.482392 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.482324 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flm8g\" (UniqueName: \"kubernetes.io/projected/b77f6b18-984e-450e-a62b-92795158af88-kube-api-access-flm8g\") pod \"maas-api-key-cleanup-29889675-tt8ks\" (UID: \"b77f6b18-984e-450e-a62b-92795158af88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" Oct 30 17:15:00.482563 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.482537 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlrxb\" (UniqueName: \"kubernetes.io/projected/880daa36-57b6-4055-ab40-224cbbbe9f5d-kube-api-access-wlrxb\") pod \"maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r\" (UID: \"880daa36-57b6-4055-ab40-224cbbbe9f5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" Oct 30 17:15:00.482639 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.482614 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhgld\" (UniqueName: \"kubernetes.io/projected/7cafb0bf-0a01-4b90-a658-9966e1a6d155-kube-api-access-hhgld\") pod \"maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b\" (UID: \"7cafb0bf-0a01-4b90-a658-9966e1a6d155\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" Oct 30 17:15:00.559307 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.559269 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbpts\" (UniqueName: \"kubernetes.io/projected/945a95c0-1afa-41a8-b23f-80db41dd13a7-kube-api-access-dbpts\") pod \"maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp\" (UID: \"945a95c0-1afa-41a8-b23f-80db41dd13a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" Oct 30 17:15:00.559490 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.559316 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/945a95c0-1afa-41a8-b23f-80db41dd13a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp\" (UID: \"945a95c0-1afa-41a8-b23f-80db41dd13a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" Oct 30 17:15:00.560041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.560017 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/945a95c0-1afa-41a8-b23f-80db41dd13a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp\" (UID: \"945a95c0-1afa-41a8-b23f-80db41dd13a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" Oct 30 17:15:00.584995 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.584972 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbpts\" (UniqueName: \"kubernetes.io/projected/945a95c0-1afa-41a8-b23f-80db41dd13a7-kube-api-access-dbpts\") pod \"maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp\" (UID: \"945a95c0-1afa-41a8-b23f-80db41dd13a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" Oct 30 17:15:00.595412 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.595390 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" Oct 30 17:15:00.604617 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.604593 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" Oct 30 17:15:00.609814 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.609790 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" Oct 30 17:15:00.628403 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:00.628232 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" Oct 30 17:15:01.055450 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:01.055411 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b"] Oct 30 17:15:01.055602 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:15:01.055486 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cafb0bf_0a01_4b90_a658_9966e1a6d155.slice/crio-e93d51e582c648dc1db45f1a22f12fa9f30e04771be0ebcf46a63ee8f20418e6 WatchSource:0}: Error finding container e93d51e582c648dc1db45f1a22f12fa9f30e04771be0ebcf46a63ee8f20418e6: Status 404 returned error can't find the container with id e93d51e582c648dc1db45f1a22f12fa9f30e04771be0ebcf46a63ee8f20418e6 Oct 30 17:15:01.100248 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:15:01.100221 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945a95c0_1afa_41a8_b23f_80db41dd13a7.slice/crio-808c693e9631ddae49e124c33fcde206dfa8b72ac0c65077b1464f6004534c81 WatchSource:0}: Error finding container 808c693e9631ddae49e124c33fcde206dfa8b72ac0c65077b1464f6004534c81: Status 404 returned error can't find the container with id 808c693e9631ddae49e124c33fcde206dfa8b72ac0c65077b1464f6004534c81 Oct 30 17:15:01.101391 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:01.101370 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp"] Oct 30 17:15:01.126631 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:01.126601 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r"] Oct 30 17:15:01.141076 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:01.141049 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks"] Oct 30 17:15:01.146780 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:01.146750 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" event={"ID":"7cafb0bf-0a01-4b90-a658-9966e1a6d155","Type":"ContainerStarted","Data":"e93d51e582c648dc1db45f1a22f12fa9f30e04771be0ebcf46a63ee8f20418e6"} Oct 30 17:15:01.148346 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:01.148323 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" event={"ID":"945a95c0-1afa-41a8-b23f-80db41dd13a7","Type":"ContainerStarted","Data":"808c693e9631ddae49e124c33fcde206dfa8b72ac0c65077b1464f6004534c81"} Oct 30 17:15:01.204398 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:15:01.204370 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880daa36_57b6_4055_ab40_224cbbbe9f5d.slice/crio-04b90a5bfcfb716716bc450581e3db4540487d94a82f87b4b405a563e1a9c0c9 WatchSource:0}: Error finding container 04b90a5bfcfb716716bc450581e3db4540487d94a82f87b4b405a563e1a9c0c9: Status 404 returned error can't find the container with id 04b90a5bfcfb716716bc450581e3db4540487d94a82f87b4b405a563e1a9c0c9 Oct 30 17:15:01.205587 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:15:01.205552 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77f6b18_984e_450e_a62b_92795158af88.slice/crio-f19b4dbdb73c9cd9c93d7e4c68a66c21379d3bbcca0d0b0c32405890e36d1849 WatchSource:0}: Error finding container f19b4dbdb73c9cd9c93d7e4c68a66c21379d3bbcca0d0b0c32405890e36d1849: Status 404 returned error can't find the container with id f19b4dbdb73c9cd9c93d7e4c68a66c21379d3bbcca0d0b0c32405890e36d1849 Oct 30 17:15:02.113274 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:02.113245 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:15:02.159417 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:02.159379 4861 generic.go:356] "Generic (PLEG): container finished" podID="7cafb0bf-0a01-4b90-a658-9966e1a6d155" containerID="5cd549e812c3d39a678ac25915d0065e8da3f948900508e6082aa54347bf2464" exitCode=60 Oct 30 17:15:02.159601 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:02.159468 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" event={"ID":"7cafb0bf-0a01-4b90-a658-9966e1a6d155","Type":"ContainerDied","Data":"5cd549e812c3d39a678ac25915d0065e8da3f948900508e6082aa54347bf2464"} Oct 30 17:15:02.159784 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:02.159766 4861 scope.go:122] "RemoveContainer" containerID="5cd549e812c3d39a678ac25915d0065e8da3f948900508e6082aa54347bf2464" Oct 30 17:15:02.162116 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:02.162088 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" event={"ID":"880daa36-57b6-4055-ab40-224cbbbe9f5d","Type":"ContainerStarted","Data":"ac2f290128686cc87fbb9b7c931006f64a0e84e4a430c6068314fe9f541557f6"} Oct 30 17:15:02.162221 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:02.162141 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" event={"ID":"880daa36-57b6-4055-ab40-224cbbbe9f5d","Type":"ContainerStarted","Data":"04b90a5bfcfb716716bc450581e3db4540487d94a82f87b4b405a563e1a9c0c9"} Oct 30 17:15:02.164696 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:02.164613 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" event={"ID":"b77f6b18-984e-450e-a62b-92795158af88","Type":"ContainerStarted","Data":"c4adf3ae93c3facf942de13ac54563f94165493d42545003ce685bfbc6e95e39"} Oct 30 17:15:02.164696 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:02.164646 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" event={"ID":"b77f6b18-984e-450e-a62b-92795158af88","Type":"ContainerStarted","Data":"f19b4dbdb73c9cd9c93d7e4c68a66c21379d3bbcca0d0b0c32405890e36d1849"} Oct 30 17:15:02.167056 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:02.167017 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" event={"ID":"945a95c0-1afa-41a8-b23f-80db41dd13a7","Type":"ContainerStarted","Data":"1db7b946d0f78ce112050b85267bc65344544552f0e9ca16e32490b292e7d9e2"} Oct 30 17:15:02.261353 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:02.261302 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" podStartSLOduration=2.261286973 podStartE2EDuration="2.261286973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.25721771 +0000 UTC m=+3132.599891751" watchObservedRunningTime="2026-10-30 17:15:02.261286973 +0000 UTC m=+3132.603961018" Oct 30 17:15:02.312299 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:02.312236 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" podStartSLOduration=2.312217845 podStartE2EDuration="2.312217845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.303312361 +0000 UTC m=+3132.645986400" watchObservedRunningTime="2026-10-30 17:15:02.312217845 +0000 UTC m=+3132.654891884" Oct 30 17:15:02.347584 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:02.347527 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" podStartSLOduration=2.347507714 podStartE2EDuration="2.347507714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:02.345926996 +0000 UTC m=+3132.688601046" watchObservedRunningTime="2026-10-30 17:15:02.347507714 +0000 UTC m=+3132.690181755" Oct 30 17:15:03.176894 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:03.176801 4861 generic.go:356] "Generic (PLEG): container finished" podID="945a95c0-1afa-41a8-b23f-80db41dd13a7" containerID="1db7b946d0f78ce112050b85267bc65344544552f0e9ca16e32490b292e7d9e2" exitCode=60 Oct 30 17:15:03.177289 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:03.176885 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" event={"ID":"945a95c0-1afa-41a8-b23f-80db41dd13a7","Type":"ContainerDied","Data":"1db7b946d0f78ce112050b85267bc65344544552f0e9ca16e32490b292e7d9e2"} Oct 30 17:15:03.177289 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:03.177173 4861 scope.go:122] "RemoveContainer" containerID="1db7b946d0f78ce112050b85267bc65344544552f0e9ca16e32490b292e7d9e2" Oct 30 17:15:03.179151 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:03.179090 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" event={"ID":"7cafb0bf-0a01-4b90-a658-9966e1a6d155","Type":"ContainerStarted","Data":"87c3ef70ee5ee765e2046dd4e635ddb9c86f2e94e41a092cd9d0694985754b1a"} Oct 30 17:15:03.181046 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:03.181021 4861 generic.go:356] "Generic (PLEG): container finished" podID="880daa36-57b6-4055-ab40-224cbbbe9f5d" containerID="ac2f290128686cc87fbb9b7c931006f64a0e84e4a430c6068314fe9f541557f6" exitCode=60 Oct 30 17:15:03.181267 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:03.181243 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" event={"ID":"880daa36-57b6-4055-ab40-224cbbbe9f5d","Type":"ContainerDied","Data":"ac2f290128686cc87fbb9b7c931006f64a0e84e4a430c6068314fe9f541557f6"} Oct 30 17:15:03.181491 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:03.181480 4861 scope.go:122] "RemoveContainer" containerID="ac2f290128686cc87fbb9b7c931006f64a0e84e4a430c6068314fe9f541557f6" Oct 30 17:15:03.183477 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:03.183452 4861 generic.go:356] "Generic (PLEG): container finished" podID="b77f6b18-984e-450e-a62b-92795158af88" containerID="c4adf3ae93c3facf942de13ac54563f94165493d42545003ce685bfbc6e95e39" exitCode=60 Oct 30 17:15:03.183591 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:03.183530 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" event={"ID":"b77f6b18-984e-450e-a62b-92795158af88","Type":"ContainerDied","Data":"c4adf3ae93c3facf942de13ac54563f94165493d42545003ce685bfbc6e95e39"} Oct 30 17:15:03.183821 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:03.183803 4861 scope.go:122] "RemoveContainer" containerID="c4adf3ae93c3facf942de13ac54563f94165493d42545003ce685bfbc6e95e39" Oct 30 17:15:03.267439 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:03.267389 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" podStartSLOduration=3.267376899 podStartE2EDuration="3.267376899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.265018468 +0000 UTC m=+3133.607692506" watchObservedRunningTime="2026-10-30 17:15:03.267376899 +0000 UTC m=+3133.610050937" Oct 30 17:15:04.195389 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:04.195297 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" event={"ID":"880daa36-57b6-4055-ab40-224cbbbe9f5d","Type":"ContainerStarted","Data":"22a5eaa3cf973e9ebeca33789a47586f59258d97975be45e17c15abf3e662fa6"} Oct 30 17:15:04.197778 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:04.197751 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" event={"ID":"b77f6b18-984e-450e-a62b-92795158af88","Type":"ContainerStarted","Data":"4fea65e101c2d089b7076e42aa64aff662c5b7bc79af49a6fbbda9586b76cd0b"} Oct 30 17:15:04.199899 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:04.199866 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" event={"ID":"945a95c0-1afa-41a8-b23f-80db41dd13a7","Type":"ContainerStarted","Data":"83535ba063dab413aa8750e6561b25c8c4d72b1199322164301e84653f89a618"} Oct 30 17:15:04.202767 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:04.202023 4861 generic.go:356] "Generic (PLEG): container finished" podID="7cafb0bf-0a01-4b90-a658-9966e1a6d155" containerID="87c3ef70ee5ee765e2046dd4e635ddb9c86f2e94e41a092cd9d0694985754b1a" exitCode=60 Oct 30 17:15:04.202767 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:04.202062 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" event={"ID":"7cafb0bf-0a01-4b90-a658-9966e1a6d155","Type":"ContainerDied","Data":"87c3ef70ee5ee765e2046dd4e635ddb9c86f2e94e41a092cd9d0694985754b1a"} Oct 30 17:15:04.202767 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:04.202107 4861 scope.go:122] "RemoveContainer" containerID="5cd549e812c3d39a678ac25915d0065e8da3f948900508e6082aa54347bf2464" Oct 30 17:15:04.202767 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:04.202407 4861 scope.go:122] "RemoveContainer" containerID="87c3ef70ee5ee765e2046dd4e635ddb9c86f2e94e41a092cd9d0694985754b1a" Oct 30 17:15:04.202767 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:15:04.202711 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b_odh-ai-gateway-infra(7cafb0bf-0a01-4b90-a658-9966e1a6d155)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" podUID="7cafb0bf-0a01-4b90-a658-9966e1a6d155" Oct 30 17:15:05.213336 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:05.213295 4861 generic.go:356] "Generic (PLEG): container finished" podID="b77f6b18-984e-450e-a62b-92795158af88" containerID="4fea65e101c2d089b7076e42aa64aff662c5b7bc79af49a6fbbda9586b76cd0b" exitCode=60 Oct 30 17:15:05.213789 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:05.213579 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" event={"ID":"b77f6b18-984e-450e-a62b-92795158af88","Type":"ContainerDied","Data":"4fea65e101c2d089b7076e42aa64aff662c5b7bc79af49a6fbbda9586b76cd0b"} Oct 30 17:15:05.213789 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:05.213687 4861 scope.go:122] "RemoveContainer" containerID="c4adf3ae93c3facf942de13ac54563f94165493d42545003ce685bfbc6e95e39" Oct 30 17:15:05.213891 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:05.213816 4861 scope.go:122] "RemoveContainer" containerID="4fea65e101c2d089b7076e42aa64aff662c5b7bc79af49a6fbbda9586b76cd0b" Oct 30 17:15:05.214229 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:15:05.214204 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889675-tt8ks_odh-ai-gateway-infra(b77f6b18-984e-450e-a62b-92795158af88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" podUID="b77f6b18-984e-450e-a62b-92795158af88" Oct 30 17:15:05.216216 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:05.216192 4861 generic.go:356] "Generic (PLEG): container finished" podID="945a95c0-1afa-41a8-b23f-80db41dd13a7" containerID="83535ba063dab413aa8750e6561b25c8c4d72b1199322164301e84653f89a618" exitCode=60 Oct 30 17:15:05.216296 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:05.216270 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" event={"ID":"945a95c0-1afa-41a8-b23f-80db41dd13a7","Type":"ContainerDied","Data":"83535ba063dab413aa8750e6561b25c8c4d72b1199322164301e84653f89a618"} Oct 30 17:15:05.216543 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:05.216523 4861 scope.go:122] "RemoveContainer" containerID="83535ba063dab413aa8750e6561b25c8c4d72b1199322164301e84653f89a618" Oct 30 17:15:05.216895 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:15:05.216869 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp_odh-ai-gateway-infra(945a95c0-1afa-41a8-b23f-80db41dd13a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" podUID="945a95c0-1afa-41a8-b23f-80db41dd13a7" Oct 30 17:15:05.218177 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:05.218159 4861 scope.go:122] "RemoveContainer" containerID="87c3ef70ee5ee765e2046dd4e635ddb9c86f2e94e41a092cd9d0694985754b1a" Oct 30 17:15:05.218497 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:15:05.218477 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b_odh-ai-gateway-infra(7cafb0bf-0a01-4b90-a658-9966e1a6d155)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" podUID="7cafb0bf-0a01-4b90-a658-9966e1a6d155" Oct 30 17:15:05.220136 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:05.220118 4861 generic.go:356] "Generic (PLEG): container finished" podID="880daa36-57b6-4055-ab40-224cbbbe9f5d" containerID="22a5eaa3cf973e9ebeca33789a47586f59258d97975be45e17c15abf3e662fa6" exitCode=60 Oct 30 17:15:05.220232 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:05.220206 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" event={"ID":"880daa36-57b6-4055-ab40-224cbbbe9f5d","Type":"ContainerDied","Data":"22a5eaa3cf973e9ebeca33789a47586f59258d97975be45e17c15abf3e662fa6"} Oct 30 17:15:05.220390 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:05.220375 4861 scope.go:122] "RemoveContainer" containerID="22a5eaa3cf973e9ebeca33789a47586f59258d97975be45e17c15abf3e662fa6" Oct 30 17:15:05.220662 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:15:05.220627 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r_odh-ai-gateway-infra(880daa36-57b6-4055-ab40-224cbbbe9f5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" podUID="880daa36-57b6-4055-ab40-224cbbbe9f5d" Oct 30 17:15:05.229330 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:05.229311 4861 scope.go:122] "RemoveContainer" containerID="1db7b946d0f78ce112050b85267bc65344544552f0e9ca16e32490b292e7d9e2" Oct 30 17:15:05.246663 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:05.246346 4861 scope.go:122] "RemoveContainer" containerID="ac2f290128686cc87fbb9b7c931006f64a0e84e4a430c6068314fe9f541557f6" Oct 30 17:15:06.230617 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:06.230587 4861 scope.go:122] "RemoveContainer" containerID="83535ba063dab413aa8750e6561b25c8c4d72b1199322164301e84653f89a618" Oct 30 17:15:06.231181 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:15:06.231030 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp_odh-ai-gateway-infra(945a95c0-1afa-41a8-b23f-80db41dd13a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" podUID="945a95c0-1afa-41a8-b23f-80db41dd13a7" Oct 30 17:15:06.232834 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:06.232809 4861 scope.go:122] "RemoveContainer" containerID="22a5eaa3cf973e9ebeca33789a47586f59258d97975be45e17c15abf3e662fa6" Oct 30 17:15:06.233210 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:15:06.233185 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r_odh-ai-gateway-infra(880daa36-57b6-4055-ab40-224cbbbe9f5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" podUID="880daa36-57b6-4055-ab40-224cbbbe9f5d" Oct 30 17:15:06.234662 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:06.234641 4861 scope.go:122] "RemoveContainer" containerID="4fea65e101c2d089b7076e42aa64aff662c5b7bc79af49a6fbbda9586b76cd0b" Oct 30 17:15:06.235055 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:15:06.235031 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889675-tt8ks_odh-ai-gateway-infra(b77f6b18-984e-450e-a62b-92795158af88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" podUID="b77f6b18-984e-450e-a62b-92795158af88" Oct 30 17:15:13.735363 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:13.735326 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799"] Oct 30 17:15:13.741666 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:13.741647 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:15:13.765351 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:13.765324 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799"] Oct 30 17:15:13.771876 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:13.771846 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e15d1f-serving-cert\"" Oct 30 17:15:13.818209 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:13.818162 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c432eb4-2ac7-424b-9a55-6d21233b4807-maas-api-tls\") pod \"maas-api-e2e-inf-a-e15d1f-67db6b66-z5799\" (UID: \"3c432eb4-2ac7-424b-9a55-6d21233b4807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:15:13.818398 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:13.818334 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c432eb4-2ac7-424b-9a55-6d21233b4807-metrics-certs\") pod \"maas-api-e2e-inf-a-e15d1f-67db6b66-z5799\" (UID: \"3c432eb4-2ac7-424b-9a55-6d21233b4807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:15:13.818487 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:13.818466 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fjv5\" (UniqueName: \"kubernetes.io/projected/3c432eb4-2ac7-424b-9a55-6d21233b4807-kube-api-access-9fjv5\") pod \"maas-api-e2e-inf-a-e15d1f-67db6b66-z5799\" (UID: \"3c432eb4-2ac7-424b-9a55-6d21233b4807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:15:13.818551 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:13.818530 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c432eb4-2ac7-424b-9a55-6d21233b4807-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e15d1f-67db6b66-z5799\" (UID: \"3c432eb4-2ac7-424b-9a55-6d21233b4807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:15:13.920030 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:13.919998 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c432eb4-2ac7-424b-9a55-6d21233b4807-maas-api-tls\") pod \"maas-api-e2e-inf-a-e15d1f-67db6b66-z5799\" (UID: \"3c432eb4-2ac7-424b-9a55-6d21233b4807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:15:13.920204 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:13.920097 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c432eb4-2ac7-424b-9a55-6d21233b4807-metrics-certs\") pod \"maas-api-e2e-inf-a-e15d1f-67db6b66-z5799\" (UID: \"3c432eb4-2ac7-424b-9a55-6d21233b4807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:15:13.920204 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:13.920163 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fjv5\" (UniqueName: \"kubernetes.io/projected/3c432eb4-2ac7-424b-9a55-6d21233b4807-kube-api-access-9fjv5\") pod \"maas-api-e2e-inf-a-e15d1f-67db6b66-z5799\" (UID: \"3c432eb4-2ac7-424b-9a55-6d21233b4807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:15:13.920334 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:13.920291 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c432eb4-2ac7-424b-9a55-6d21233b4807-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e15d1f-67db6b66-z5799\" (UID: \"3c432eb4-2ac7-424b-9a55-6d21233b4807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:15:13.920965 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:13.920940 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c432eb4-2ac7-424b-9a55-6d21233b4807-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e15d1f-67db6b66-z5799\" (UID: \"3c432eb4-2ac7-424b-9a55-6d21233b4807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:15:13.922834 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:13.922810 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c432eb4-2ac7-424b-9a55-6d21233b4807-maas-api-tls\") pod \"maas-api-e2e-inf-a-e15d1f-67db6b66-z5799\" (UID: \"3c432eb4-2ac7-424b-9a55-6d21233b4807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:15:13.922834 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:13.922816 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c432eb4-2ac7-424b-9a55-6d21233b4807-metrics-certs\") pod \"maas-api-e2e-inf-a-e15d1f-67db6b66-z5799\" (UID: \"3c432eb4-2ac7-424b-9a55-6d21233b4807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:15:13.947043 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:13.947021 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fjv5\" (UniqueName: \"kubernetes.io/projected/3c432eb4-2ac7-424b-9a55-6d21233b4807-kube-api-access-9fjv5\") pod \"maas-api-e2e-inf-a-e15d1f-67db6b66-z5799\" (UID: \"3c432eb4-2ac7-424b-9a55-6d21233b4807\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:15:14.057451 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:14.057371 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:15:14.375464 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:15:14.375436 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c432eb4_2ac7_424b_9a55_6d21233b4807.slice/crio-976d83bd59eeaf5879bd8726008a45c14b34633096b81f7affdd9e16d7125651 WatchSource:0}: Error finding container 976d83bd59eeaf5879bd8726008a45c14b34633096b81f7affdd9e16d7125651: Status 404 returned error can't find the container with id 976d83bd59eeaf5879bd8726008a45c14b34633096b81f7affdd9e16d7125651 Oct 30 17:15:14.375672 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:14.375651 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799"] Oct 30 17:15:14.401482 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:14.401446 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" event={"ID":"3c432eb4-2ac7-424b-9a55-6d21233b4807","Type":"ContainerStarted","Data":"976d83bd59eeaf5879bd8726008a45c14b34633096b81f7affdd9e16d7125651"} Oct 30 17:15:15.411102 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:15.411065 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" event={"ID":"3c432eb4-2ac7-424b-9a55-6d21233b4807","Type":"ContainerStarted","Data":"9fb0fe9310c16121d51c80b91edd77cf760a844a5becf46d7f96c183016e5a76"} Oct 30 17:15:15.411670 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:15.411167 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:15:15.471108 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:15.471045 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" podStartSLOduration=2.097703518 podStartE2EDuration="2.471027545s" totalImagesPullingTime="373.324027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:15.469320273 +0000 UTC m=+3145.811994311" watchObservedRunningTime="2026-10-30 17:15:15.471027545 +0000 UTC m=+3145.813701584" Oct 30 17:15:15.801772 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:15.800037 4861 scope.go:122] "RemoveContainer" containerID="87c3ef70ee5ee765e2046dd4e635ddb9c86f2e94e41a092cd9d0694985754b1a" Oct 30 17:15:16.425090 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:16.425055 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" event={"ID":"7cafb0bf-0a01-4b90-a658-9966e1a6d155","Type":"ContainerStarted","Data":"5dda93111ca3fb9c574222f3254be9d3a50577f1907d9043dca0bfe781141092"} Oct 30 17:15:16.800355 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:16.800327 4861 scope.go:122] "RemoveContainer" containerID="22a5eaa3cf973e9ebeca33789a47586f59258d97975be45e17c15abf3e662fa6" Oct 30 17:15:16.800624 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:16.800599 4861 scope.go:122] "RemoveContainer" containerID="4fea65e101c2d089b7076e42aa64aff662c5b7bc79af49a6fbbda9586b76cd0b" Oct 30 17:15:17.435097 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:17.435004 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" event={"ID":"880daa36-57b6-4055-ab40-224cbbbe9f5d","Type":"ContainerStarted","Data":"16214e88f210f96fbf2d224d1b18521f1b0d3d4cf67acb8855d89cf97901c49d"} Oct 30 17:15:17.437340 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:17.437311 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" event={"ID":"b77f6b18-984e-450e-a62b-92795158af88","Type":"ContainerStarted","Data":"11e8e40ea706054ffe503666974f02e6de415459633175a39bfc3399bb7d42ad"} Oct 30 17:15:17.439432 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:17.439409 4861 generic.go:356] "Generic (PLEG): container finished" podID="7cafb0bf-0a01-4b90-a658-9966e1a6d155" containerID="5dda93111ca3fb9c574222f3254be9d3a50577f1907d9043dca0bfe781141092" exitCode=60 Oct 30 17:15:17.439547 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:17.439461 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" event={"ID":"7cafb0bf-0a01-4b90-a658-9966e1a6d155","Type":"ContainerDied","Data":"5dda93111ca3fb9c574222f3254be9d3a50577f1907d9043dca0bfe781141092"} Oct 30 17:15:17.439547 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:17.439500 4861 scope.go:122] "RemoveContainer" containerID="87c3ef70ee5ee765e2046dd4e635ddb9c86f2e94e41a092cd9d0694985754b1a" Oct 30 17:15:17.439871 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:17.439853 4861 scope.go:122] "RemoveContainer" containerID="5dda93111ca3fb9c574222f3254be9d3a50577f1907d9043dca0bfe781141092" Oct 30 17:15:17.440209 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:15:17.440187 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b_odh-ai-gateway-infra(7cafb0bf-0a01-4b90-a658-9966e1a6d155)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" podUID="7cafb0bf-0a01-4b90-a658-9966e1a6d155" Oct 30 17:15:17.482801 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:17.482781 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b"] Oct 30 17:15:17.836745 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:17.835934 4861 scope.go:122] "RemoveContainer" containerID="83535ba063dab413aa8750e6561b25c8c4d72b1199322164301e84653f89a618" Oct 30 17:15:18.449327 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.449243 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" event={"ID":"945a95c0-1afa-41a8-b23f-80db41dd13a7","Type":"ContainerStarted","Data":"085387797b1ce695bee7944d017e999af1bc32c9d9ddd4dee466f31fb519cd58"} Oct 30 17:15:18.453495 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.453469 4861 generic.go:356] "Generic (PLEG): container finished" podID="880daa36-57b6-4055-ab40-224cbbbe9f5d" containerID="16214e88f210f96fbf2d224d1b18521f1b0d3d4cf67acb8855d89cf97901c49d" exitCode=60 Oct 30 17:15:18.453616 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.453540 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" event={"ID":"880daa36-57b6-4055-ab40-224cbbbe9f5d","Type":"ContainerDied","Data":"16214e88f210f96fbf2d224d1b18521f1b0d3d4cf67acb8855d89cf97901c49d"} Oct 30 17:15:18.453616 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.453599 4861 scope.go:122] "RemoveContainer" containerID="22a5eaa3cf973e9ebeca33789a47586f59258d97975be45e17c15abf3e662fa6" Oct 30 17:15:18.454102 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.454081 4861 scope.go:122] "RemoveContainer" containerID="16214e88f210f96fbf2d224d1b18521f1b0d3d4cf67acb8855d89cf97901c49d" Oct 30 17:15:18.454446 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:15:18.454419 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r_odh-ai-gateway-infra(880daa36-57b6-4055-ab40-224cbbbe9f5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" podUID="880daa36-57b6-4055-ab40-224cbbbe9f5d" Oct 30 17:15:18.455841 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.455818 4861 generic.go:356] "Generic (PLEG): container finished" podID="b77f6b18-984e-450e-a62b-92795158af88" containerID="11e8e40ea706054ffe503666974f02e6de415459633175a39bfc3399bb7d42ad" exitCode=60 Oct 30 17:15:18.455912 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.455848 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" event={"ID":"b77f6b18-984e-450e-a62b-92795158af88","Type":"ContainerDied","Data":"11e8e40ea706054ffe503666974f02e6de415459633175a39bfc3399bb7d42ad"} Oct 30 17:15:18.456161 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.456147 4861 scope.go:122] "RemoveContainer" containerID="11e8e40ea706054ffe503666974f02e6de415459633175a39bfc3399bb7d42ad" Oct 30 17:15:18.456404 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:15:18.456387 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889675-tt8ks_odh-ai-gateway-infra(b77f6b18-984e-450e-a62b-92795158af88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" podUID="b77f6b18-984e-450e-a62b-92795158af88" Oct 30 17:15:18.470453 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.470434 4861 scope.go:122] "RemoveContainer" containerID="4fea65e101c2d089b7076e42aa64aff662c5b7bc79af49a6fbbda9586b76cd0b" Oct 30 17:15:18.493568 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.493542 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r"] Oct 30 17:15:18.582038 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.581972 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks"] Oct 30 17:15:18.660336 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.660303 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" Oct 30 17:15:18.753374 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.753344 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cafb0bf-0a01-4b90-a658-9966e1a6d155-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cafb0bf-0a01-4b90-a658-9966e1a6d155-maas-api-service-ca\") pod \"7cafb0bf-0a01-4b90-a658-9966e1a6d155\" (UID: \"7cafb0bf-0a01-4b90-a658-9966e1a6d155\") " Oct 30 17:15:18.753707 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.753679 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cafb0bf-0a01-4b90-a658-9966e1a6d155-kube-api-access-hhgld\" (UniqueName: \"kubernetes.io/projected/7cafb0bf-0a01-4b90-a658-9966e1a6d155-kube-api-access-hhgld\") pod \"7cafb0bf-0a01-4b90-a658-9966e1a6d155\" (UID: \"7cafb0bf-0a01-4b90-a658-9966e1a6d155\") " Oct 30 17:15:18.753895 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.753801 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cafb0bf-0a01-4b90-a658-9966e1a6d155-maas-api-service-ca" pod "7cafb0bf-0a01-4b90-a658-9966e1a6d155" (UID: "7cafb0bf-0a01-4b90-a658-9966e1a6d155"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:18.754628 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.754602 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cafb0bf-0a01-4b90-a658-9966e1a6d155-maas-api-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:15:18.756850 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.756829 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cafb0bf-0a01-4b90-a658-9966e1a6d155-kube-api-access-hhgld" pod "7cafb0bf-0a01-4b90-a658-9966e1a6d155" (UID: "7cafb0bf-0a01-4b90-a658-9966e1a6d155"). InnerVolumeSpecName "kube-api-access-hhgld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:18.856486 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:18.856455 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhgld\" (UniqueName: \"kubernetes.io/projected/7cafb0bf-0a01-4b90-a658-9966e1a6d155-kube-api-access-hhgld\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.465608 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.465578 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" Oct 30 17:15:19.466169 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.465611 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b" event={"ID":"7cafb0bf-0a01-4b90-a658-9966e1a6d155","Type":"ContainerDied","Data":"e93d51e582c648dc1db45f1a22f12fa9f30e04771be0ebcf46a63ee8f20418e6"} Oct 30 17:15:19.466169 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.465687 4861 scope.go:122] "RemoveContainer" containerID="5dda93111ca3fb9c574222f3254be9d3a50577f1907d9043dca0bfe781141092" Oct 30 17:15:19.473107 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.473077 4861 generic.go:356] "Generic (PLEG): container finished" podID="945a95c0-1afa-41a8-b23f-80db41dd13a7" containerID="085387797b1ce695bee7944d017e999af1bc32c9d9ddd4dee466f31fb519cd58" exitCode=60 Oct 30 17:15:19.473223 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.473204 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" event={"ID":"945a95c0-1afa-41a8-b23f-80db41dd13a7","Type":"ContainerDied","Data":"085387797b1ce695bee7944d017e999af1bc32c9d9ddd4dee466f31fb519cd58"} Oct 30 17:15:19.473643 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.473608 4861 scope.go:122] "RemoveContainer" containerID="085387797b1ce695bee7944d017e999af1bc32c9d9ddd4dee466f31fb519cd58" Oct 30 17:15:19.474058 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:15:19.474032 4861 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp_odh-ai-gateway-infra(945a95c0-1afa-41a8-b23f-80db41dd13a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" podUID="945a95c0-1afa-41a8-b23f-80db41dd13a7" Oct 30 17:15:19.481559 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.481541 4861 scope.go:122] "RemoveContainer" containerID="83535ba063dab413aa8750e6561b25c8c4d72b1199322164301e84653f89a618" Oct 30 17:15:19.506484 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.506450 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp"] Oct 30 17:15:19.527825 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.527793 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b"] Oct 30 17:15:19.537318 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.537291 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-9bb5b1-29889675-hhj9b"] Oct 30 17:15:19.766032 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.765995 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" Oct 30 17:15:19.807021 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.806983 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" Oct 30 17:15:19.819536 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.819495 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cafb0bf-0a01-4b90-a658-9966e1a6d155" path="/var/lib/kubelet/pods/7cafb0bf-0a01-4b90-a658-9966e1a6d155/volumes" Oct 30 17:15:19.878895 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.878859 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b77f6b18-984e-450e-a62b-92795158af88-kube-api-access-flm8g\" (UniqueName: \"kubernetes.io/projected/b77f6b18-984e-450e-a62b-92795158af88-kube-api-access-flm8g\") pod \"b77f6b18-984e-450e-a62b-92795158af88\" (UID: \"b77f6b18-984e-450e-a62b-92795158af88\") " Oct 30 17:15:19.879142 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.878920 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880daa36-57b6-4055-ab40-224cbbbe9f5d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/880daa36-57b6-4055-ab40-224cbbbe9f5d-maas-api-service-ca\") pod \"880daa36-57b6-4055-ab40-224cbbbe9f5d\" (UID: \"880daa36-57b6-4055-ab40-224cbbbe9f5d\") " Oct 30 17:15:19.879142 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.879007 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b77f6b18-984e-450e-a62b-92795158af88-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77f6b18-984e-450e-a62b-92795158af88-maas-api-service-ca\") pod \"b77f6b18-984e-450e-a62b-92795158af88\" (UID: \"b77f6b18-984e-450e-a62b-92795158af88\") " Oct 30 17:15:19.879263 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.879252 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880daa36-57b6-4055-ab40-224cbbbe9f5d-kube-api-access-wlrxb\" (UniqueName: \"kubernetes.io/projected/880daa36-57b6-4055-ab40-224cbbbe9f5d-kube-api-access-wlrxb\") pod \"880daa36-57b6-4055-ab40-224cbbbe9f5d\" (UID: \"880daa36-57b6-4055-ab40-224cbbbe9f5d\") " Oct 30 17:15:19.879340 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.879311 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880daa36-57b6-4055-ab40-224cbbbe9f5d-maas-api-service-ca" pod "880daa36-57b6-4055-ab40-224cbbbe9f5d" (UID: "880daa36-57b6-4055-ab40-224cbbbe9f5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:19.879405 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.879357 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b77f6b18-984e-450e-a62b-92795158af88-maas-api-service-ca" pod "b77f6b18-984e-450e-a62b-92795158af88" (UID: "b77f6b18-984e-450e-a62b-92795158af88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:19.879952 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.879928 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/880daa36-57b6-4055-ab40-224cbbbe9f5d-maas-api-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.879952 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.879954 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77f6b18-984e-450e-a62b-92795158af88-maas-api-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.881758 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.881699 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77f6b18-984e-450e-a62b-92795158af88-kube-api-access-flm8g" pod "b77f6b18-984e-450e-a62b-92795158af88" (UID: "b77f6b18-984e-450e-a62b-92795158af88"). InnerVolumeSpecName "kube-api-access-flm8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:19.882078 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.882051 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880daa36-57b6-4055-ab40-224cbbbe9f5d-kube-api-access-wlrxb" pod "880daa36-57b6-4055-ab40-224cbbbe9f5d" (UID: "880daa36-57b6-4055-ab40-224cbbbe9f5d"). InnerVolumeSpecName "kube-api-access-wlrxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:19.981599 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.981530 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlrxb\" (UniqueName: \"kubernetes.io/projected/880daa36-57b6-4055-ab40-224cbbbe9f5d-kube-api-access-wlrxb\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:15:19.981599 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:19.981558 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flm8g\" (UniqueName: \"kubernetes.io/projected/b77f6b18-984e-450e-a62b-92795158af88-kube-api-access-flm8g\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.291488 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:15:20.291402 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:15:20.484115 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.484082 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" Oct 30 17:15:20.484644 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.484116 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks" event={"ID":"b77f6b18-984e-450e-a62b-92795158af88","Type":"ContainerDied","Data":"f19b4dbdb73c9cd9c93d7e4c68a66c21379d3bbcca0d0b0c32405890e36d1849"} Oct 30 17:15:20.484644 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.484186 4861 scope.go:122] "RemoveContainer" containerID="11e8e40ea706054ffe503666974f02e6de415459633175a39bfc3399bb7d42ad" Oct 30 17:15:20.491848 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.491825 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" Oct 30 17:15:20.491973 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.491878 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r" event={"ID":"880daa36-57b6-4055-ab40-224cbbbe9f5d","Type":"ContainerDied","Data":"04b90a5bfcfb716716bc450581e3db4540487d94a82f87b4b405a563e1a9c0c9"} Oct 30 17:15:20.498870 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.498841 4861 scope.go:122] "RemoveContainer" containerID="16214e88f210f96fbf2d224d1b18521f1b0d3d4cf67acb8855d89cf97901c49d" Oct 30 17:15:20.591752 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.591682 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks"] Oct 30 17:15:20.596810 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.596781 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889675-tt8ks"] Oct 30 17:15:20.647816 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.647787 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r"] Oct 30 17:15:20.652705 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.652679 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a77b48-29889675-mrp7r"] Oct 30 17:15:20.680867 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.680847 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" Oct 30 17:15:20.793805 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.793765 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945a95c0-1afa-41a8-b23f-80db41dd13a7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/945a95c0-1afa-41a8-b23f-80db41dd13a7-maas-api-service-ca\") pod \"945a95c0-1afa-41a8-b23f-80db41dd13a7\" (UID: \"945a95c0-1afa-41a8-b23f-80db41dd13a7\") " Oct 30 17:15:20.793994 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.793827 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945a95c0-1afa-41a8-b23f-80db41dd13a7-kube-api-access-dbpts\" (UniqueName: \"kubernetes.io/projected/945a95c0-1afa-41a8-b23f-80db41dd13a7-kube-api-access-dbpts\") pod \"945a95c0-1afa-41a8-b23f-80db41dd13a7\" (UID: \"945a95c0-1afa-41a8-b23f-80db41dd13a7\") " Oct 30 17:15:20.794233 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.794205 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945a95c0-1afa-41a8-b23f-80db41dd13a7-maas-api-service-ca" pod "945a95c0-1afa-41a8-b23f-80db41dd13a7" (UID: "945a95c0-1afa-41a8-b23f-80db41dd13a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:20.794547 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.794521 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/945a95c0-1afa-41a8-b23f-80db41dd13a7-maas-api-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.796375 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.796348 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945a95c0-1afa-41a8-b23f-80db41dd13a7-kube-api-access-dbpts" pod "945a95c0-1afa-41a8-b23f-80db41dd13a7" (UID: "945a95c0-1afa-41a8-b23f-80db41dd13a7"). InnerVolumeSpecName "kube-api-access-dbpts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:20.896344 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:20.896309 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbpts\" (UniqueName: \"kubernetes.io/projected/945a95c0-1afa-41a8-b23f-80db41dd13a7-kube-api-access-dbpts\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.431411 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:21.431379 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:15:21.536960 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:21.536913 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" event={"ID":"945a95c0-1afa-41a8-b23f-80db41dd13a7","Type":"ContainerDied","Data":"808c693e9631ddae49e124c33fcde206dfa8b72ac0c65077b1464f6004534c81"} Oct 30 17:15:21.537447 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:21.537001 4861 scope.go:122] "RemoveContainer" containerID="085387797b1ce695bee7944d017e999af1bc32c9d9ddd4dee466f31fb519cd58" Oct 30 17:15:21.537447 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:21.537118 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp" Oct 30 17:15:21.615387 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:21.615358 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp"] Oct 30 17:15:21.619986 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:21.619959 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6e08bd-29889675-kwqxp"] Oct 30 17:15:21.811119 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:21.811083 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="880daa36-57b6-4055-ab40-224cbbbe9f5d" path="/var/lib/kubelet/pods/880daa36-57b6-4055-ab40-224cbbbe9f5d/volumes" Oct 30 17:15:21.811613 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:21.811598 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945a95c0-1afa-41a8-b23f-80db41dd13a7" path="/var/lib/kubelet/pods/945a95c0-1afa-41a8-b23f-80db41dd13a7/volumes" Oct 30 17:15:21.812094 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:21.812079 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b77f6b18-984e-450e-a62b-92795158af88" path="/var/lib/kubelet/pods/b77f6b18-984e-450e-a62b-92795158af88/volumes" Oct 30 17:15:48.992732 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:48.992695 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk"] Oct 30 17:15:50.291085 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:15:50.291036 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:15:53.994001 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:53.993960 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" podUID="a2e800b2-30e2-421e-a5db-9a16b6c346f6" containerName="payload-processing" containerID="cri-o://304feb67af22bad1d1411ef5d00db789890fed0c5be514570f5275802fd0a46b" gracePeriod=25 Oct 30 17:15:54.498851 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:54.498824 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:15:54.554465 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:54.554428 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e800b2-30e2-421e-a5db-9a16b6c346f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e800b2-30e2-421e-a5db-9a16b6c346f6-plugins-config-volume\") pod \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\" (UID: \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\") " Oct 30 17:15:54.555538 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:54.555508 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e800b2-30e2-421e-a5db-9a16b6c346f6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2e800b2-30e2-421e-a5db-9a16b6c346f6-routing-overlay-volume\") pod \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\" (UID: \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\") " Oct 30 17:15:54.555694 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:54.555640 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e800b2-30e2-421e-a5db-9a16b6c346f6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2e800b2-30e2-421e-a5db-9a16b6c346f6-provider-credentials\") pod \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\" (UID: \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\") " Oct 30 17:15:54.555826 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:54.555788 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e800b2-30e2-421e-a5db-9a16b6c346f6-kube-api-access-hrcj5\" (UniqueName: \"kubernetes.io/projected/a2e800b2-30e2-421e-a5db-9a16b6c346f6-kube-api-access-hrcj5\") pod \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\" (UID: \"a2e800b2-30e2-421e-a5db-9a16b6c346f6\") " Oct 30 17:15:54.556081 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:54.556054 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e800b2-30e2-421e-a5db-9a16b6c346f6-plugins-config-volume" pod "a2e800b2-30e2-421e-a5db-9a16b6c346f6" (UID: "a2e800b2-30e2-421e-a5db-9a16b6c346f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:54.557028 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:54.556501 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2e800b2-30e2-421e-a5db-9a16b6c346f6-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:15:54.557028 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:54.556886 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e800b2-30e2-421e-a5db-9a16b6c346f6-routing-overlay-volume" pod "a2e800b2-30e2-421e-a5db-9a16b6c346f6" (UID: "a2e800b2-30e2-421e-a5db-9a16b6c346f6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:54.559967 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:54.559770 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e800b2-30e2-421e-a5db-9a16b6c346f6-provider-credentials" pod "a2e800b2-30e2-421e-a5db-9a16b6c346f6" (UID: "a2e800b2-30e2-421e-a5db-9a16b6c346f6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:54.560587 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:54.560562 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e800b2-30e2-421e-a5db-9a16b6c346f6-kube-api-access-hrcj5" pod "a2e800b2-30e2-421e-a5db-9a16b6c346f6" (UID: "a2e800b2-30e2-421e-a5db-9a16b6c346f6"). InnerVolumeSpecName "kube-api-access-hrcj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:54.657947 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:54.657909 4861 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a2e800b2-30e2-421e-a5db-9a16b6c346f6-routing-overlay-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:15:54.657947 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:54.657946 4861 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a2e800b2-30e2-421e-a5db-9a16b6c346f6-provider-credentials\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:15:54.658168 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:54.657961 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrcj5\" (UniqueName: \"kubernetes.io/projected/a2e800b2-30e2-421e-a5db-9a16b6c346f6-kube-api-access-hrcj5\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:15:55.038809 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:55.036544 4861 generic.go:356] "Generic (PLEG): container finished" podID="a2e800b2-30e2-421e-a5db-9a16b6c346f6" containerID="304feb67af22bad1d1411ef5d00db789890fed0c5be514570f5275802fd0a46b" exitCode=0 Oct 30 17:15:55.038809 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:55.036661 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" event={"ID":"a2e800b2-30e2-421e-a5db-9a16b6c346f6","Type":"ContainerDied","Data":"304feb67af22bad1d1411ef5d00db789890fed0c5be514570f5275802fd0a46b"} Oct 30 17:15:55.038809 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:55.036699 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" event={"ID":"a2e800b2-30e2-421e-a5db-9a16b6c346f6","Type":"ContainerDied","Data":"a1d49163aad88a84fe288b9858d826c307f68fc1354500af1dc7d50a52d1e846"} Oct 30 17:15:55.038809 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:55.036739 4861 scope.go:122] "RemoveContainer" containerID="304feb67af22bad1d1411ef5d00db789890fed0c5be514570f5275802fd0a46b" Oct 30 17:15:55.038809 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:55.036884 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk" Oct 30 17:15:55.054947 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:55.054925 4861 scope.go:122] "RemoveContainer" containerID="304feb67af22bad1d1411ef5d00db789890fed0c5be514570f5275802fd0a46b" Oct 30 17:15:55.055283 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:15:55.055260 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"304feb67af22bad1d1411ef5d00db789890fed0c5be514570f5275802fd0a46b\": container with ID starting with 304feb67af22bad1d1411ef5d00db789890fed0c5be514570f5275802fd0a46b not found: ID does not exist" containerID="304feb67af22bad1d1411ef5d00db789890fed0c5be514570f5275802fd0a46b" Oct 30 17:15:55.055382 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:55.055289 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304feb67af22bad1d1411ef5d00db789890fed0c5be514570f5275802fd0a46b"} err="failed to get container status \"304feb67af22bad1d1411ef5d00db789890fed0c5be514570f5275802fd0a46b\": rpc error: code = NotFound desc = could not find container \"304feb67af22bad1d1411ef5d00db789890fed0c5be514570f5275802fd0a46b\": container with ID starting with 304feb67af22bad1d1411ef5d00db789890fed0c5be514570f5275802fd0a46b not found: ID does not exist" Oct 30 17:15:55.187829 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:55.178316 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk"] Oct 30 17:15:55.187829 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:55.184453 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-f94fb5/payload-processing-external-model-e2e-extmod-a-f94fb5-9bd9b7twk"] Oct 30 17:15:55.820153 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:55.820101 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e800b2-30e2-421e-a5db-9a16b6c346f6" path="/var/lib/kubelet/pods/a2e800b2-30e2-421e-a5db-9a16b6c346f6/volumes" Oct 30 17:15:58.020972 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.020933 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm"] Oct 30 17:15:58.023029 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.022997 4861 cpu_manager.go:414] "containerMap: removing container" podUID="945a95c0-1afa-41a8-b23f-80db41dd13a7" containerName="cleanup" Oct 30 17:15:58.023183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023035 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945a95c0-1afa-41a8-b23f-80db41dd13a7" containerName="cleanup" Oct 30 17:15:58.023183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023061 4861 cpu_manager.go:414] "containerMap: removing container" podUID="a2e800b2-30e2-421e-a5db-9a16b6c346f6" containerName="payload-processing" Oct 30 17:15:58.023183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023069 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e800b2-30e2-421e-a5db-9a16b6c346f6" containerName="payload-processing" Oct 30 17:15:58.023183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023094 4861 cpu_manager.go:414] "containerMap: removing container" podUID="945a95c0-1afa-41a8-b23f-80db41dd13a7" containerName="cleanup" Oct 30 17:15:58.023183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023102 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945a95c0-1afa-41a8-b23f-80db41dd13a7" containerName="cleanup" Oct 30 17:15:58.023183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023116 4861 cpu_manager.go:414] "containerMap: removing container" podUID="7cafb0bf-0a01-4b90-a658-9966e1a6d155" containerName="cleanup" Oct 30 17:15:58.023183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023128 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cafb0bf-0a01-4b90-a658-9966e1a6d155" containerName="cleanup" Oct 30 17:15:58.023183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023144 4861 cpu_manager.go:414] "containerMap: removing container" podUID="880daa36-57b6-4055-ab40-224cbbbe9f5d" containerName="cleanup" Oct 30 17:15:58.023183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023150 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880daa36-57b6-4055-ab40-224cbbbe9f5d" containerName="cleanup" Oct 30 17:15:58.023183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023170 4861 cpu_manager.go:414] "containerMap: removing container" podUID="880daa36-57b6-4055-ab40-224cbbbe9f5d" containerName="cleanup" Oct 30 17:15:58.023183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023177 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880daa36-57b6-4055-ab40-224cbbbe9f5d" containerName="cleanup" Oct 30 17:15:58.023749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023226 4861 cpu_manager.go:414] "containerMap: removing container" podUID="7cafb0bf-0a01-4b90-a658-9966e1a6d155" containerName="cleanup" Oct 30 17:15:58.023749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023233 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cafb0bf-0a01-4b90-a658-9966e1a6d155" containerName="cleanup" Oct 30 17:15:58.023749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023263 4861 cpu_manager.go:414] "containerMap: removing container" podUID="7cafb0bf-0a01-4b90-a658-9966e1a6d155" containerName="cleanup" Oct 30 17:15:58.023749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023269 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cafb0bf-0a01-4b90-a658-9966e1a6d155" containerName="cleanup" Oct 30 17:15:58.023749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023303 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b77f6b18-984e-450e-a62b-92795158af88" containerName="cleanup" Oct 30 17:15:58.023749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023310 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77f6b18-984e-450e-a62b-92795158af88" containerName="cleanup" Oct 30 17:15:58.023749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023336 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b77f6b18-984e-450e-a62b-92795158af88" containerName="cleanup" Oct 30 17:15:58.023749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023343 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77f6b18-984e-450e-a62b-92795158af88" containerName="cleanup" Oct 30 17:15:58.023749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023388 4861 cpu_manager.go:414] "containerMap: removing container" podUID="945a95c0-1afa-41a8-b23f-80db41dd13a7" containerName="cleanup" Oct 30 17:15:58.023749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023395 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945a95c0-1afa-41a8-b23f-80db41dd13a7" containerName="cleanup" Oct 30 17:15:58.024467 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023897 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cafb0bf-0a01-4b90-a658-9966e1a6d155" containerName="cleanup" Oct 30 17:15:58.024467 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023939 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77f6b18-984e-450e-a62b-92795158af88" containerName="cleanup" Oct 30 17:15:58.024467 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023949 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77f6b18-984e-450e-a62b-92795158af88" containerName="cleanup" Oct 30 17:15:58.024467 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023971 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="945a95c0-1afa-41a8-b23f-80db41dd13a7" containerName="cleanup" Oct 30 17:15:58.024467 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.023981 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="880daa36-57b6-4055-ab40-224cbbbe9f5d" containerName="cleanup" Oct 30 17:15:58.024467 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.024003 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="945a95c0-1afa-41a8-b23f-80db41dd13a7" containerName="cleanup" Oct 30 17:15:58.024467 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.024012 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="880daa36-57b6-4055-ab40-224cbbbe9f5d" containerName="cleanup" Oct 30 17:15:58.024467 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.024029 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e800b2-30e2-421e-a5db-9a16b6c346f6" containerName="payload-processing" Oct 30 17:15:58.024467 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.024048 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="880daa36-57b6-4055-ab40-224cbbbe9f5d" containerName="cleanup" Oct 30 17:15:58.024467 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.024067 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cafb0bf-0a01-4b90-a658-9966e1a6d155" containerName="cleanup" Oct 30 17:15:58.046829 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.046797 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm"] Oct 30 17:15:58.047013 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.046916 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" Oct 30 17:15:58.073799 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.073770 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bbc36f\"" Oct 30 17:15:58.119078 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.119047 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bbc36f-dockercfg-6j6ch\"" Oct 30 17:15:58.296510 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.296428 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvdrw\" (UniqueName: \"kubernetes.io/projected/9345c3c1-50db-406b-81b6-202cb9b75496-kube-api-access-wvdrw\") pod \"payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm\" (UID: \"9345c3c1-50db-406b-81b6-202cb9b75496\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" Oct 30 17:15:58.297141 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.297115 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9345c3c1-50db-406b-81b6-202cb9b75496-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm\" (UID: \"9345c3c1-50db-406b-81b6-202cb9b75496\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" Oct 30 17:15:58.399359 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.399316 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvdrw\" (UniqueName: \"kubernetes.io/projected/9345c3c1-50db-406b-81b6-202cb9b75496-kube-api-access-wvdrw\") pod \"payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm\" (UID: \"9345c3c1-50db-406b-81b6-202cb9b75496\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" Oct 30 17:15:58.399757 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.399703 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9345c3c1-50db-406b-81b6-202cb9b75496-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm\" (UID: \"9345c3c1-50db-406b-81b6-202cb9b75496\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" Oct 30 17:15:58.400477 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.400452 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9345c3c1-50db-406b-81b6-202cb9b75496-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm\" (UID: \"9345c3c1-50db-406b-81b6-202cb9b75496\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" Oct 30 17:15:58.425022 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.424998 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvdrw\" (UniqueName: \"kubernetes.io/projected/9345c3c1-50db-406b-81b6-202cb9b75496-kube-api-access-wvdrw\") pod \"payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm\" (UID: \"9345c3c1-50db-406b-81b6-202cb9b75496\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" Oct 30 17:15:58.675569 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:58.675469 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" Oct 30 17:15:59.005994 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.005962 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm"] Oct 30 17:15:59.006261 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:15:59.006234 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9345c3c1_50db_406b_81b6_202cb9b75496.slice/crio-da3bdbc9b324ce9173aaf8b46a7da3ed1fbe25156ada1c5aac1dda6a2f81067c WatchSource:0}: Error finding container da3bdbc9b324ce9173aaf8b46a7da3ed1fbe25156ada1c5aac1dda6a2f81067c: Status 404 returned error can't find the container with id da3bdbc9b324ce9173aaf8b46a7da3ed1fbe25156ada1c5aac1dda6a2f81067c Oct 30 17:15:59.087830 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.087780 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" event={"ID":"9345c3c1-50db-406b-81b6-202cb9b75496","Type":"ContainerStarted","Data":"da3bdbc9b324ce9173aaf8b46a7da3ed1fbe25156ada1c5aac1dda6a2f81067c"} Oct 30 17:15:59.686417 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.686379 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt"] Oct 30 17:15:59.687398 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.687376 4861 cpu_manager.go:414] "containerMap: removing container" podUID="880daa36-57b6-4055-ab40-224cbbbe9f5d" containerName="cleanup" Oct 30 17:15:59.687398 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.687399 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880daa36-57b6-4055-ab40-224cbbbe9f5d" containerName="cleanup" Oct 30 17:15:59.687574 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.687449 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b77f6b18-984e-450e-a62b-92795158af88" containerName="cleanup" Oct 30 17:15:59.687574 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.687456 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77f6b18-984e-450e-a62b-92795158af88" containerName="cleanup" Oct 30 17:15:59.687639 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.687601 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cafb0bf-0a01-4b90-a658-9966e1a6d155" containerName="cleanup" Oct 30 17:15:59.687639 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.687613 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77f6b18-984e-450e-a62b-92795158af88" containerName="cleanup" Oct 30 17:15:59.687639 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.687621 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="945a95c0-1afa-41a8-b23f-80db41dd13a7" containerName="cleanup" Oct 30 17:15:59.691881 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.691861 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" Oct 30 17:15:59.716065 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.715803 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt"] Oct 30 17:15:59.720840 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.720816 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4c6596\"" Oct 30 17:15:59.743354 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.743319 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c"] Oct 30 17:15:59.752193 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.752169 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" Oct 30 17:15:59.770361 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.770327 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4c6596-dockercfg-j6vs4\"" Oct 30 17:15:59.776862 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.776825 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c"] Oct 30 17:15:59.838087 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.838011 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56c9a33-1b9e-48dc-9ebe-d80778b20e35-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c\" (UID: \"b56c9a33-1b9e-48dc-9ebe-d80778b20e35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" Oct 30 17:15:59.838274 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.838125 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334f60fa-9e62-417d-8965-9a5a0f5bff47-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt\" (UID: \"334f60fa-9e62-417d-8965-9a5a0f5bff47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" Oct 30 17:15:59.838274 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.838163 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g8kk\" (UniqueName: \"kubernetes.io/projected/334f60fa-9e62-417d-8965-9a5a0f5bff47-kube-api-access-6g8kk\") pod \"payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt\" (UID: \"334f60fa-9e62-417d-8965-9a5a0f5bff47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" Oct 30 17:15:59.838274 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.838214 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhbpl\" (UniqueName: \"kubernetes.io/projected/b56c9a33-1b9e-48dc-9ebe-d80778b20e35-kube-api-access-vhbpl\") pod \"payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c\" (UID: \"b56c9a33-1b9e-48dc-9ebe-d80778b20e35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" Oct 30 17:15:59.944795 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.941543 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56c9a33-1b9e-48dc-9ebe-d80778b20e35-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c\" (UID: \"b56c9a33-1b9e-48dc-9ebe-d80778b20e35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" Oct 30 17:15:59.944795 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.941670 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334f60fa-9e62-417d-8965-9a5a0f5bff47-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt\" (UID: \"334f60fa-9e62-417d-8965-9a5a0f5bff47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" Oct 30 17:15:59.944795 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.941711 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g8kk\" (UniqueName: \"kubernetes.io/projected/334f60fa-9e62-417d-8965-9a5a0f5bff47-kube-api-access-6g8kk\") pod \"payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt\" (UID: \"334f60fa-9e62-417d-8965-9a5a0f5bff47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" Oct 30 17:15:59.944795 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.941789 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbpl\" (UniqueName: \"kubernetes.io/projected/b56c9a33-1b9e-48dc-9ebe-d80778b20e35-kube-api-access-vhbpl\") pod \"payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c\" (UID: \"b56c9a33-1b9e-48dc-9ebe-d80778b20e35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" Oct 30 17:15:59.944795 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.943466 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56c9a33-1b9e-48dc-9ebe-d80778b20e35-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c\" (UID: \"b56c9a33-1b9e-48dc-9ebe-d80778b20e35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" Oct 30 17:15:59.944795 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:15:59.944296 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334f60fa-9e62-417d-8965-9a5a0f5bff47-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt\" (UID: \"334f60fa-9e62-417d-8965-9a5a0f5bff47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" Oct 30 17:16:00.025744 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:00.020685 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbpl\" (UniqueName: \"kubernetes.io/projected/b56c9a33-1b9e-48dc-9ebe-d80778b20e35-kube-api-access-vhbpl\") pod \"payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c\" (UID: \"b56c9a33-1b9e-48dc-9ebe-d80778b20e35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" Oct 30 17:16:00.042752 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:00.041131 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g8kk\" (UniqueName: \"kubernetes.io/projected/334f60fa-9e62-417d-8965-9a5a0f5bff47-kube-api-access-6g8kk\") pod \"payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt\" (UID: \"334f60fa-9e62-417d-8965-9a5a0f5bff47\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" Oct 30 17:16:00.087057 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:00.087019 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" Oct 30 17:16:00.142777 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:00.142713 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" event={"ID":"9345c3c1-50db-406b-81b6-202cb9b75496","Type":"ContainerStarted","Data":"f8ccfedb4595751530406be630724798a6bb2aca4dcf15d9769b5e00aca079ed"} Oct 30 17:16:00.153863 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:00.153830 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" Oct 30 17:16:00.201159 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:00.201032 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" podStartSLOduration=3.201013346 podStartE2EDuration="3.201013346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:00.200066738 +0000 UTC m=+3190.542740777" watchObservedRunningTime="2026-10-30 17:16:00.201013346 +0000 UTC m=+3190.543687384" Oct 30 17:16:00.310913 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:00.310876 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" Oct 30 17:16:00.512768 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:16:00.507014 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56c9a33_1b9e_48dc_9ebe_d80778b20e35.slice/crio-e8f5486d8b05d5640f54fa6876fdc1adc2528c8ada258575035dbffacbd4cabd WatchSource:0}: Error finding container e8f5486d8b05d5640f54fa6876fdc1adc2528c8ada258575035dbffacbd4cabd: Status 404 returned error can't find the container with id e8f5486d8b05d5640f54fa6876fdc1adc2528c8ada258575035dbffacbd4cabd Oct 30 17:16:00.512768 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:00.509579 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c"] Oct 30 17:16:00.782748 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:00.781309 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt"] Oct 30 17:16:00.804943 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:16:00.804909 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334f60fa_9e62_417d_8965_9a5a0f5bff47.slice/crio-ae709990ae00512e3b40b629b20816285b56fa2954890c6e000344625d1d5bda WatchSource:0}: Error finding container ae709990ae00512e3b40b629b20816285b56fa2954890c6e000344625d1d5bda: Status 404 returned error can't find the container with id ae709990ae00512e3b40b629b20816285b56fa2954890c6e000344625d1d5bda Oct 30 17:16:01.154759 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:01.154624 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" event={"ID":"334f60fa-9e62-417d-8965-9a5a0f5bff47","Type":"ContainerStarted","Data":"216bfe1202745b08df5791a3b127631d5815543355955ef426ac21b06972e5d1"} Oct 30 17:16:01.154759 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:01.154698 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" event={"ID":"334f60fa-9e62-417d-8965-9a5a0f5bff47","Type":"ContainerStarted","Data":"ae709990ae00512e3b40b629b20816285b56fa2954890c6e000344625d1d5bda"} Oct 30 17:16:01.155276 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:01.154776 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" Oct 30 17:16:01.157558 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:01.157527 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" event={"ID":"b56c9a33-1b9e-48dc-9ebe-d80778b20e35","Type":"ContainerStarted","Data":"433c4aee9dd83f4d3f9fb1c11446498cf3298240b667ac24a851257979ede425"} Oct 30 17:16:01.157708 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:01.157581 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" event={"ID":"b56c9a33-1b9e-48dc-9ebe-d80778b20e35","Type":"ContainerStarted","Data":"e8f5486d8b05d5640f54fa6876fdc1adc2528c8ada258575035dbffacbd4cabd"} Oct 30 17:16:01.157708 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:01.157610 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" Oct 30 17:16:01.212883 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:01.212818 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" podStartSLOduration=2.212798878 podStartE2EDuration="2.212798878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:01.20742017 +0000 UTC m=+3191.550094209" watchObservedRunningTime="2026-10-30 17:16:01.212798878 +0000 UTC m=+3191.555472916" Oct 30 17:16:01.255924 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:01.255855 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" podStartSLOduration=2.255836328 podStartE2EDuration="2.255836328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:01.254198992 +0000 UTC m=+3191.596873030" watchObservedRunningTime="2026-10-30 17:16:01.255836328 +0000 UTC m=+3191.598510367" Oct 30 17:16:02.167284 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:02.167182 4861 generic.go:356] "Generic (PLEG): container finished" podID="ce90a319-7468-4c37-8065-01f08642df9d" containerID="de35be27d45cdd31e9edbed0c278c82b2c295ef510c523af0ef06e4ddeb0f1db" exitCode=0 Oct 30 17:16:02.167788 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:02.167298 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" event={"ID":"ce90a319-7468-4c37-8065-01f08642df9d","Type":"ContainerDied","Data":"de35be27d45cdd31e9edbed0c278c82b2c295ef510c523af0ef06e4ddeb0f1db"} Oct 30 17:16:02.168170 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:02.168146 4861 scope.go:122] "RemoveContainer" containerID="de35be27d45cdd31e9edbed0c278c82b2c295ef510c523af0ef06e4ddeb0f1db" Oct 30 17:16:02.170428 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:02.170390 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:16:02.170582 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:02.170485 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:16:02.170582 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:02.170541 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:16:03.178628 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:03.178587 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" event={"ID":"ce90a319-7468-4c37-8065-01f08642df9d","Type":"ContainerStarted","Data":"9d5615a42ba0f48d5b3ddbe7a8f9f3e530fd819bb75e12c004371ec74da69ad5"} Oct 30 17:16:03.281675 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:03.281639 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:16:03.283353 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:03.283326 4861 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 17:16:03.283487 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:03.283372 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" podUID="ce90a319-7468-4c37-8065-01f08642df9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 17:16:03.423749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:03.420998 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:16:04.281715 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:04.281666 4861 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 17:16:04.282368 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:04.281742 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" podUID="ce90a319-7468-4c37-8065-01f08642df9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 17:16:05.282265 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:05.282230 4861 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 17:16:05.282701 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:05.282273 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" podUID="ce90a319-7468-4c37-8065-01f08642df9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 17:16:06.282475 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:06.282444 4861 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 17:16:06.282922 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:06.282493 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" podUID="ce90a319-7468-4c37-8065-01f08642df9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 17:16:07.282007 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:07.281972 4861 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 17:16:07.282193 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:07.282026 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" podUID="ce90a319-7468-4c37-8065-01f08642df9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 17:16:08.281805 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:08.281771 4861 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 17:16:08.282255 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:08.281822 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" podUID="ce90a319-7468-4c37-8065-01f08642df9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 17:16:09.287029 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:09.286999 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:16:09.287981 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:09.287958 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:16:12.171434 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:12.171391 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" Oct 30 17:16:12.171989 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:12.171574 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" Oct 30 17:16:12.172055 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:12.172028 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" Oct 30 17:16:14.841608 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:14.841572 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj"] Oct 30 17:16:14.852159 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:14.852131 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:14.878586 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:14.878544 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj"] Oct 30 17:16:14.885009 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:14.884988 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-85eb87-openshift-default-dockercfg-csf7x\"" Oct 30 17:16:15.098843 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.077612 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-credential-socket\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.099234 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.099203 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-podinfo\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.099393 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.099378 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7dd7b91-6cba-4446-9f47-c536520520bf-istiod-ca-cert\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.099513 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.099499 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-workload-socket\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.099677 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.099660 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-data\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.118258 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.118221 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-envoy\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.120232 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.118606 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mzmm\" (UniqueName: \"kubernetes.io/projected/f7dd7b91-6cba-4446-9f47-c536520520bf-kube-api-access-5mzmm\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.120782 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.120758 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-workload-certs\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.121021 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.121004 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-token\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.236765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.228860 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-token\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.236765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.229017 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-credential-socket\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.236765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.229041 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-podinfo\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.236765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.229103 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7dd7b91-6cba-4446-9f47-c536520520bf-istiod-ca-cert\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.236765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.229146 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-workload-socket\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.236765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.229196 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-data\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.236765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.229406 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-envoy\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.236765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.229559 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mzmm\" (UniqueName: \"kubernetes.io/projected/f7dd7b91-6cba-4446-9f47-c536520520bf-kube-api-access-5mzmm\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.236765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.229627 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-workload-certs\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.236765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.230605 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-workload-certs\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.236765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.231802 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-credential-socket\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.236765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.235482 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-podinfo\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.236765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.235890 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-workload-socket\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.236765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.236453 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-data\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.237464 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.237440 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7dd7b91-6cba-4446-9f47-c536520520bf-istiod-ca-cert\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.241059 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.241031 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-envoy\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.259854 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.259805 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-token\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.263987 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.263948 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mzmm\" (UniqueName: \"kubernetes.io/projected/f7dd7b91-6cba-4446-9f47-c536520520bf-kube-api-access-5mzmm\") pod \"e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.470313 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.470284 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:15.817509 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:16:15.817481 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7dd7b91_6cba_4446_9f47_c536520520bf.slice/crio-f750e41005dc475168523f5edb0aa9f52253e03ceea1d69d8d7e889879a35cb0 WatchSource:0}: Error finding container f750e41005dc475168523f5edb0aa9f52253e03ceea1d69d8d7e889879a35cb0: Status 404 returned error can't find the container with id f750e41005dc475168523f5edb0aa9f52253e03ceea1d69d8d7e889879a35cb0 Oct 30 17:16:15.818834 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.818793 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj"] Oct 30 17:16:15.821104 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.821069 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:16:15.821228 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.821138 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:16:15.821228 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.821170 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:16:15.882594 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.882568 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf"] Oct 30 17:16:15.889048 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.889028 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" Oct 30 17:16:15.907937 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.907911 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d80ebb-dockercfg-2zpmm\"" Oct 30 17:16:15.909092 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.909069 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d80ebb\"" Oct 30 17:16:15.910448 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:15.909960 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf"] Oct 30 17:16:16.070804 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:16.064459 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0682eb07-3174-4f1f-af2c-a455e9ddc740-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf\" (UID: \"0682eb07-3174-4f1f-af2c-a455e9ddc740\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" Oct 30 17:16:16.070804 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:16.064512 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgvhq\" (UniqueName: \"kubernetes.io/projected/0682eb07-3174-4f1f-af2c-a455e9ddc740-kube-api-access-mgvhq\") pod \"payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf\" (UID: \"0682eb07-3174-4f1f-af2c-a455e9ddc740\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" Oct 30 17:16:16.170867 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:16.166423 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0682eb07-3174-4f1f-af2c-a455e9ddc740-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf\" (UID: \"0682eb07-3174-4f1f-af2c-a455e9ddc740\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" Oct 30 17:16:16.170867 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:16.166480 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgvhq\" (UniqueName: \"kubernetes.io/projected/0682eb07-3174-4f1f-af2c-a455e9ddc740-kube-api-access-mgvhq\") pod \"payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf\" (UID: \"0682eb07-3174-4f1f-af2c-a455e9ddc740\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" Oct 30 17:16:16.170867 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:16.169821 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0682eb07-3174-4f1f-af2c-a455e9ddc740-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf\" (UID: \"0682eb07-3174-4f1f-af2c-a455e9ddc740\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" Oct 30 17:16:16.252778 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:16.251505 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgvhq\" (UniqueName: \"kubernetes.io/projected/0682eb07-3174-4f1f-af2c-a455e9ddc740-kube-api-access-mgvhq\") pod \"payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf\" (UID: \"0682eb07-3174-4f1f-af2c-a455e9ddc740\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" Oct 30 17:16:16.346197 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:16.346097 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" event={"ID":"f7dd7b91-6cba-4446-9f47-c536520520bf","Type":"ContainerStarted","Data":"ab7536ff7fc96f49373a624bcad025886f0c58be4ecab65fddef24ffdef42a04"} Oct 30 17:16:16.346197 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:16.346166 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" event={"ID":"f7dd7b91-6cba-4446-9f47-c536520520bf","Type":"ContainerStarted","Data":"f750e41005dc475168523f5edb0aa9f52253e03ceea1d69d8d7e889879a35cb0"} Oct 30 17:16:16.403667 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:16.403601 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" podStartSLOduration=2.403577871 podStartE2EDuration="2.403577871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:16.397403615 +0000 UTC m=+3206.740077671" watchObservedRunningTime="2026-10-30 17:16:16.403577871 +0000 UTC m=+3206.746251910" Oct 30 17:16:16.471409 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:16.471374 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:16.473212 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:16.473185 4861 patch_prober.go:29] interesting pod/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 17:16:16.473344 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:16.473235 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" podUID="f7dd7b91-6cba-4446-9f47-c536520520bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 17:16:16.515776 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:16.513832 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" Oct 30 17:16:16.998567 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:16:16.998530 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0682eb07_3174_4f1f_af2c_a455e9ddc740.slice/crio-b38e63f717a0e594f024e47647d6aa3e2c6be7c3df03e4e574397db7effe3e5e WatchSource:0}: Error finding container b38e63f717a0e594f024e47647d6aa3e2c6be7c3df03e4e574397db7effe3e5e: Status 404 returned error can't find the container with id b38e63f717a0e594f024e47647d6aa3e2c6be7c3df03e4e574397db7effe3e5e Oct 30 17:16:16.998944 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:16.998636 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf"] Oct 30 17:16:17.357667 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:17.357571 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" event={"ID":"0682eb07-3174-4f1f-af2c-a455e9ddc740","Type":"ContainerStarted","Data":"cb7dfbeacea581abc053734fda8571a19acc94a573409b4238cf58e6139deecc"} Oct 30 17:16:17.357667 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:17.357638 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" event={"ID":"0682eb07-3174-4f1f-af2c-a455e9ddc740","Type":"ContainerStarted","Data":"b38e63f717a0e594f024e47647d6aa3e2c6be7c3df03e4e574397db7effe3e5e"} Oct 30 17:16:17.357922 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:17.357817 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" Oct 30 17:16:17.406231 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:17.406182 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" podStartSLOduration=2.406168931 podStartE2EDuration="2.406168931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:16:17.40149943 +0000 UTC m=+3207.744173469" watchObservedRunningTime="2026-10-30 17:16:17.406168931 +0000 UTC m=+3207.748842968" Oct 30 17:16:17.475493 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:17.475468 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:18.366098 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:18.366063 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:18.367002 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:18.366980 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:16:20.292079 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:16:20.292037 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:16:28.369056 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:16:28.369025 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" Oct 30 17:16:50.291118 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:16:50.291079 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:17:11.016368 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.016331 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9"] Oct 30 17:17:11.023652 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.023631 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:11.043409 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.043381 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9"] Oct 30 17:17:11.043861 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.043838 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6254deaf-17a7-4214-928c-2d64d6da2684-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:11.044231 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.044036 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-metrics-certs\") pod \"maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:11.044231 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.044140 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-maas-api-tls\") pod \"maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:11.044231 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.044197 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bpph\" (UniqueName: \"kubernetes.io/projected/6254deaf-17a7-4214-928c-2d64d6da2684-kube-api-access-8bpph\") pod \"maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:11.050864 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.050838 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d80ebb-serving-cert\"" Oct 30 17:17:11.145479 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.145445 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-maas-api-tls\") pod \"maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:11.145699 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.145550 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bpph\" (UniqueName: \"kubernetes.io/projected/6254deaf-17a7-4214-928c-2d64d6da2684-kube-api-access-8bpph\") pod \"maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:11.145699 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:17:11.145570 4861 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-serving-cert: secret "maas-api-e2e-inf-a-d80ebb-serving-cert" not found Oct 30 17:17:11.145699 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.145614 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6254deaf-17a7-4214-928c-2d64d6da2684-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:11.145699 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:17:11.145624 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-maas-api-tls podName:6254deaf-17a7-4214-928c-2d64d6da2684 nodeName:}" failed. No retries permitted until 2026-10-30 17:17:11.645608009 +0000 UTC m=+3261.988282053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-maas-api-tls") pod "maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" (UID: "6254deaf-17a7-4214-928c-2d64d6da2684") : secret "maas-api-e2e-inf-a-d80ebb-serving-cert" not found Oct 30 17:17:11.146080 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.145900 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-metrics-certs\") pod \"maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:11.146334 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.146312 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6254deaf-17a7-4214-928c-2d64d6da2684-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:11.148601 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.148569 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-metrics-certs\") pod \"maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:11.171599 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.171571 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bpph\" (UniqueName: \"kubernetes.io/projected/6254deaf-17a7-4214-928c-2d64d6da2684-kube-api-access-8bpph\") pod \"maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:11.663833 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.663794 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-maas-api-tls\") pod \"maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:11.666416 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.666398 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-maas-api-tls\") pod \"maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:11.946862 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:11.946774 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:12.369894 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:12.369859 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9"] Oct 30 17:17:12.954824 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:12.954788 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" event={"ID":"6254deaf-17a7-4214-928c-2d64d6da2684","Type":"ContainerStarted","Data":"a79bcf3f0bc60690cf77e38a901f6ff8bd5170d064060918f1624ac02705d07e"} Oct 30 17:17:12.955014 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:12.954850 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" event={"ID":"6254deaf-17a7-4214-928c-2d64d6da2684","Type":"ContainerStarted","Data":"4c1e9263cc4a4df0322e8ab0a4147c310493ee5e4514780f9d8b0b424acdb6b1"} Oct 30 17:17:12.955014 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:12.954897 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:13.004474 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:13.004381 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" podStartSLOduration=2.644813283 podStartE2EDuration="3.004367011s" totalImagesPullingTime="359.553728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:12.999810625 +0000 UTC m=+3263.342484663" watchObservedRunningTime="2026-10-30 17:17:13.004367011 +0000 UTC m=+3263.347041049" Oct 30 17:17:16.614148 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:16.614114 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz"] Oct 30 17:17:16.619582 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:16.619560 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:17:16.638706 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:16.638669 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz"] Oct 30 17:17:16.658149 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:16.647435 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-85eb87-serving-cert\"" Oct 30 17:17:16.756673 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:16.756633 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c79376-2a02-4240-bde9-c58d20acb2f5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz\" (UID: \"a2c79376-2a02-4240-bde9-c58d20acb2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:17:16.756863 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:16.756761 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxnsj\" (UniqueName: \"kubernetes.io/projected/a2c79376-2a02-4240-bde9-c58d20acb2f5-kube-api-access-wxnsj\") pod \"maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz\" (UID: \"a2c79376-2a02-4240-bde9-c58d20acb2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:17:16.756976 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:16.756961 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c79376-2a02-4240-bde9-c58d20acb2f5-maas-api-tls\") pod \"maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz\" (UID: \"a2c79376-2a02-4240-bde9-c58d20acb2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:17:16.757033 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:16.756984 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c79376-2a02-4240-bde9-c58d20acb2f5-metrics-certs\") pod \"maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz\" (UID: \"a2c79376-2a02-4240-bde9-c58d20acb2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:17:16.865049 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:16.861270 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c79376-2a02-4240-bde9-c58d20acb2f5-maas-api-tls\") pod \"maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz\" (UID: \"a2c79376-2a02-4240-bde9-c58d20acb2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:17:16.865049 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:16.861321 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c79376-2a02-4240-bde9-c58d20acb2f5-metrics-certs\") pod \"maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz\" (UID: \"a2c79376-2a02-4240-bde9-c58d20acb2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:17:16.865049 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:16.861485 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c79376-2a02-4240-bde9-c58d20acb2f5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz\" (UID: \"a2c79376-2a02-4240-bde9-c58d20acb2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:17:16.865049 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:16.862577 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c79376-2a02-4240-bde9-c58d20acb2f5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz\" (UID: \"a2c79376-2a02-4240-bde9-c58d20acb2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:17:16.878744 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:16.877487 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxnsj\" (UniqueName: \"kubernetes.io/projected/a2c79376-2a02-4240-bde9-c58d20acb2f5-kube-api-access-wxnsj\") pod \"maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz\" (UID: \"a2c79376-2a02-4240-bde9-c58d20acb2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:17:16.902745 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:16.902627 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c79376-2a02-4240-bde9-c58d20acb2f5-maas-api-tls\") pod \"maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz\" (UID: \"a2c79376-2a02-4240-bde9-c58d20acb2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:17:16.968746 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:16.904052 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c79376-2a02-4240-bde9-c58d20acb2f5-metrics-certs\") pod \"maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz\" (UID: \"a2c79376-2a02-4240-bde9-c58d20acb2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:17:16.968746 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:16.944164 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxnsj\" (UniqueName: \"kubernetes.io/projected/a2c79376-2a02-4240-bde9-c58d20acb2f5-kube-api-access-wxnsj\") pod \"maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz\" (UID: \"a2c79376-2a02-4240-bde9-c58d20acb2f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:17:17.235366 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:17.235335 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:17:17.557441 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:17.557416 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz"] Oct 30 17:17:17.558949 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:17:17.558921 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c79376_2a02_4240_bde9_c58d20acb2f5.slice/crio-2e6b59030187b035cc2e5dd739ee98a896cfa5ebd094d0302812d14ca8510945 WatchSource:0}: Error finding container 2e6b59030187b035cc2e5dd739ee98a896cfa5ebd094d0302812d14ca8510945: Status 404 returned error can't find the container with id 2e6b59030187b035cc2e5dd739ee98a896cfa5ebd094d0302812d14ca8510945 Oct 30 17:17:18.078098 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:18.078037 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" event={"ID":"a2c79376-2a02-4240-bde9-c58d20acb2f5","Type":"ContainerStarted","Data":"2e6b59030187b035cc2e5dd739ee98a896cfa5ebd094d0302812d14ca8510945"} Oct 30 17:17:18.968009 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:18.967980 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:17:19.106031 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:19.105991 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" event={"ID":"a2c79376-2a02-4240-bde9-c58d20acb2f5","Type":"ContainerStarted","Data":"d06065b8f894a43160097a8bbcf058351b5c2a91d4e54872e89a2ef244523133"} Oct 30 17:17:19.107357 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:19.107329 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:17:20.291764 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:17:20.291694 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:17:26.130701 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:26.130671 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:17:26.178243 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:26.178192 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" podStartSLOduration=9.698983479 podStartE2EDuration="10.178179636s" totalImagesPullingTime="479.196157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:19.162115132 +0000 UTC m=+3269.504789169" watchObservedRunningTime="2026-10-30 17:17:26.178179636 +0000 UTC m=+3276.520853673" Oct 30 17:17:50.291438 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:17:50.291401 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:17:50.397743 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:50.397704 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 17:17:50.451994 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:50.451968 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 17:17:55.678458 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:55.678422 4861 generic.go:356] "Generic (PLEG): container finished" podID="f7dd7b91-6cba-4446-9f47-c536520520bf" containerID="ab7536ff7fc96f49373a624bcad025886f0c58be4ecab65fddef24ffdef42a04" exitCode=0 Oct 30 17:17:55.679024 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:55.678482 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" event={"ID":"f7dd7b91-6cba-4446-9f47-c536520520bf","Type":"ContainerDied","Data":"ab7536ff7fc96f49373a624bcad025886f0c58be4ecab65fddef24ffdef42a04"} Oct 30 17:17:55.679024 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:55.678942 4861 scope.go:122] "RemoveContainer" containerID="ab7536ff7fc96f49373a624bcad025886f0c58be4ecab65fddef24ffdef42a04" Oct 30 17:17:55.681039 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:55.681008 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:17:55.681130 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:55.681087 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:17:55.681172 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:55.681127 4861 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 17:17:56.689002 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:56.688963 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" event={"ID":"f7dd7b91-6cba-4446-9f47-c536520520bf","Type":"ContainerStarted","Data":"0c140559e9dd1e7a300298d9a534075458e0f694d018c00be59ad78d53fae585"} Oct 30 17:17:57.472289 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:57.472256 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:17:57.472586 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:57.472340 4861 patch_prober.go:29] interesting pod/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 17:17:57.472586 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:57.472390 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" podUID="f7dd7b91-6cba-4446-9f47-c536520520bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 17:17:58.471254 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:58.471222 4861 patch_prober.go:29] interesting pod/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 17:17:58.471642 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:58.471273 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" podUID="f7dd7b91-6cba-4446-9f47-c536520520bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 17:17:59.471098 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:59.471061 4861 patch_prober.go:29] interesting pod/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 17:17:59.471285 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:17:59.471119 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" podUID="f7dd7b91-6cba-4446-9f47-c536520520bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 17:18:00.471533 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:00.471498 4861 patch_prober.go:29] interesting pod/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 17:18:00.471977 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:00.471553 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" podUID="f7dd7b91-6cba-4446-9f47-c536520520bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 17:18:01.470651 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:01.470619 4861 patch_prober.go:29] interesting pod/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" start-of-body= Oct 30 17:18:01.470910 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:01.470671 4861 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" podUID="f7dd7b91-6cba-4446-9f47-c536520520bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.38:15021/healthz/ready\": dial tcp 10.132.1.38:15021: connect: connection refused" Oct 30 17:18:02.476159 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:02.476127 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:18:02.476621 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:02.476593 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:18:02.477742 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:02.477704 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:18:20.380382 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:18:20.380336 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:18:30.704745 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.704623 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8"] Oct 30 17:18:30.711481 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.711455 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.733464 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.732867 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8"] Oct 30 17:18:30.733688 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.733664 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4c6596\"/\"kube-root-ca.crt\"" Oct 30 17:18:30.733850 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.733704 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4c6596\"/\"default-dockercfg-q9xh4\"" Oct 30 17:18:30.742952 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.742927 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4c6596\"/\"test-model-4c6596-kserve-self-signed-certs\"" Oct 30 17:18:30.742952 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.742927 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4c6596\"/\"openshift-service-ca.crt\"" Oct 30 17:18:30.798477 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.798441 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-tmp-dir\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.798679 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.798492 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjkhf\" (UniqueName: \"kubernetes.io/projected/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-kube-api-access-gjkhf\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.798679 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.798661 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-model-cache\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.798902 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.798715 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-dshm\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.798902 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.798790 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-tls-certs\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.798902 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.798866 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-home\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.900147 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.900104 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-tmp-dir\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.900356 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.900179 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjkhf\" (UniqueName: \"kubernetes.io/projected/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-kube-api-access-gjkhf\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.900356 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.900268 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-model-cache\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.900356 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.900323 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-dshm\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.900599 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.900368 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-tls-certs\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.900599 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.900439 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-tmp-dir\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.900599 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.900533 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-home\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.900842 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.900669 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-model-cache\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.900917 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.900896 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-home\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.902796 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.902774 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-dshm\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.903128 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.903110 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-tls-certs\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:30.925168 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:30.925139 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjkhf\" (UniqueName: \"kubernetes.io/projected/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-kube-api-access-gjkhf\") pod \"test-model-4c6596-kserve-7f759ff96f-svdn8\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:31.033752 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:31.033624 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:31.396998 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:31.396966 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8"] Oct 30 17:18:31.397816 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:18:31.397782 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3141d3a0_8e3a_4232_b9bc_2de718b19b1e.slice/crio-1c59d4d290bbe4f0761876187276db4542ba3b7ce20e0d0c4aef9fa876759371 WatchSource:0}: Error finding container 1c59d4d290bbe4f0761876187276db4542ba3b7ce20e0d0c4aef9fa876759371: Status 404 returned error can't find the container with id 1c59d4d290bbe4f0761876187276db4542ba3b7ce20e0d0c4aef9fa876759371 Oct 30 17:18:32.142220 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:32.111793 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" event={"ID":"3141d3a0-8e3a-4232-b9bc-2de718b19b1e","Type":"ContainerStarted","Data":"d6af706071d12358749e4c562e238201ec770a0a0eb5ea74931fc3614afab20e"} Oct 30 17:18:32.142220 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:32.111857 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" event={"ID":"3141d3a0-8e3a-4232-b9bc-2de718b19b1e","Type":"ContainerStarted","Data":"1c59d4d290bbe4f0761876187276db4542ba3b7ce20e0d0c4aef9fa876759371"} Oct 30 17:18:32.171745 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:32.166831 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" podStartSLOduration=2.16681252 podStartE2EDuration="2.16681252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:32.166633306 +0000 UTC m=+3342.509307347" watchObservedRunningTime="2026-10-30 17:18:32.16681252 +0000 UTC m=+3342.509486558" Oct 30 17:18:33.034522 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:33.034487 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:33.887633 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:33.887598 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn"] Oct 30 17:18:33.893294 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:33.893269 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:33.913134 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:33.913102 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn"] Oct 30 17:18:33.920499 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:33.920476 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e15d1f\"/\"kube-root-ca.crt\"" Oct 30 17:18:33.920760 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:33.920481 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e15d1f\"/\"default-dockercfg-82jj8\"" Oct 30 17:18:33.924313 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:33.924289 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e15d1f\"/\"openshift-service-ca.crt\"" Oct 30 17:18:33.924313 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:33.924298 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e15d1f\"/\"test-model-e15d1f-kserve-self-signed-certs\"" Oct 30 17:18:33.965687 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:33.965661 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-tmp-dir\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:33.965884 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:33.965857 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1779c0b-52e6-4c97-b02c-116f12d11ff9-tls-certs\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:33.966018 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:33.965987 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-model-cache\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:33.966153 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:33.966131 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-home\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:33.966247 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:33.966234 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcw46\" (UniqueName: \"kubernetes.io/projected/e1779c0b-52e6-4c97-b02c-116f12d11ff9-kube-api-access-qcw46\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:33.966323 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:33.966304 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-dshm\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:34.068436 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:34.068399 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-tmp-dir\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:34.068637 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:34.068491 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1779c0b-52e6-4c97-b02c-116f12d11ff9-tls-certs\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:34.068637 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:34.068593 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-model-cache\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:34.068784 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:34.068684 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-home\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:34.068853 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:34.068803 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcw46\" (UniqueName: \"kubernetes.io/projected/e1779c0b-52e6-4c97-b02c-116f12d11ff9-kube-api-access-qcw46\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:34.068853 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:34.068839 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-dshm\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:34.068966 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:34.068868 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-tmp-dir\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:34.068966 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:34.068947 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-model-cache\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:34.069080 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:34.069002 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-home\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:34.071259 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:34.071229 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-dshm\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:34.071259 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:34.071249 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1779c0b-52e6-4c97-b02c-116f12d11ff9-tls-certs\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:34.105787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:34.094374 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcw46\" (UniqueName: \"kubernetes.io/projected/e1779c0b-52e6-4c97-b02c-116f12d11ff9-kube-api-access-qcw46\") pod \"test-model-e15d1f-kserve-76554c59c5-j7lsn\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:34.209626 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:34.209539 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:34.582283 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:18:34.582227 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1779c0b_52e6_4c97_b02c_116f12d11ff9.slice/crio-97f527aa5609675b0a6f3efa7a03bdf5b6f74d045ec9feb578767d37a7e9169c WatchSource:0}: Error finding container 97f527aa5609675b0a6f3efa7a03bdf5b6f74d045ec9feb578767d37a7e9169c: Status 404 returned error can't find the container with id 97f527aa5609675b0a6f3efa7a03bdf5b6f74d045ec9feb578767d37a7e9169c Oct 30 17:18:34.582761 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:34.582716 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn"] Oct 30 17:18:35.218979 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:35.218943 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" event={"ID":"e1779c0b-52e6-4c97-b02c-116f12d11ff9","Type":"ContainerStarted","Data":"ae94c9b930b2b2c124d4f5d83c22df25077915cb2f08677f97c09ba1a014be9a"} Oct 30 17:18:35.219474 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:35.218990 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" event={"ID":"e1779c0b-52e6-4c97-b02c-116f12d11ff9","Type":"ContainerStarted","Data":"97f527aa5609675b0a6f3efa7a03bdf5b6f74d045ec9feb578767d37a7e9169c"} Oct 30 17:18:35.266491 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:35.266433 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" podStartSLOduration=2.26641886 podStartE2EDuration="2.26641886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:35.262976701 +0000 UTC m=+3345.605650743" watchObservedRunningTime="2026-10-30 17:18:35.26641886 +0000 UTC m=+3345.609092901" Oct 30 17:18:36.210017 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:36.209984 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:41.034348 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:41.034310 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:41.047049 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:41.047020 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:41.288279 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:41.288191 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:18:44.210865 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:44.210832 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:44.224245 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:44.224219 4861 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:44.312857 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:18:44.312828 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:18:50.293433 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:18:50.293392 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:19:02.439168 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:02.439080 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c"] Oct 30 17:19:17.471086 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.471027 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" podUID="e73e1c71-d9da-4c54-94e7-9862e81a0be5" containerName="main" containerID="cri-o://c8e4998cf936752cab1ae1f7680ab05d920d5f70ed85600200c0f465ce2b212f" gracePeriod=45 Oct 30 17:19:17.582812 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:19:17.582781 4861 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73e1c71_d9da_4c54_94e7_9862e81a0be5.slice/crio-conmon-c8e4998cf936752cab1ae1f7680ab05d920d5f70ed85600200c0f465ce2b212f.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:19:17.692513 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.692478 4861 generic.go:356] "Generic (PLEG): container finished" podID="e73e1c71-d9da-4c54-94e7-9862e81a0be5" containerID="c8e4998cf936752cab1ae1f7680ab05d920d5f70ed85600200c0f465ce2b212f" exitCode=0 Oct 30 17:19:17.692808 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.692775 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" event={"ID":"e73e1c71-d9da-4c54-94e7-9862e81a0be5","Type":"ContainerDied","Data":"c8e4998cf936752cab1ae1f7680ab05d920d5f70ed85600200c0f465ce2b212f"} Oct 30 17:19:17.834381 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.834349 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:19:17.970430 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.970397 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-model-cache\") pod \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " Oct 30 17:19:17.970599 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.970440 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-tmp-dir\") pod \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " Oct 30 17:19:17.970599 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.970540 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-home\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-home\") pod \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " Oct 30 17:19:17.970695 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.970664 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73e1c71-d9da-4c54-94e7-9862e81a0be5-kube-api-access-whzv9\" (UniqueName: \"kubernetes.io/projected/e73e1c71-d9da-4c54-94e7-9862e81a0be5-kube-api-access-whzv9\") pod \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " Oct 30 17:19:17.970824 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.970806 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e73e1c71-d9da-4c54-94e7-9862e81a0be5-tls-certs\" (UniqueName: \"kubernetes.io/secret/e73e1c71-d9da-4c54-94e7-9862e81a0be5-tls-certs\") pod \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " Oct 30 17:19:17.970919 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.970829 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-model-cache" pod "e73e1c71-d9da-4c54-94e7-9862e81a0be5" (UID: "e73e1c71-d9da-4c54-94e7-9862e81a0be5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:17.970919 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.970876 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-dshm\") pod \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\" (UID: \"e73e1c71-d9da-4c54-94e7-9862e81a0be5\") " Oct 30 17:19:17.970919 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.970843 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-tmp-dir" pod "e73e1c71-d9da-4c54-94e7-9862e81a0be5" (UID: "e73e1c71-d9da-4c54-94e7-9862e81a0be5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:17.971116 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.970923 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-home" pod "e73e1c71-d9da-4c54-94e7-9862e81a0be5" (UID: "e73e1c71-d9da-4c54-94e7-9862e81a0be5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:17.971922 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.971890 4861 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:19:17.971922 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.971918 4861 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:19:17.972125 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.971933 4861 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:19:17.973355 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.973328 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-dshm" pod "e73e1c71-d9da-4c54-94e7-9862e81a0be5" (UID: "e73e1c71-d9da-4c54-94e7-9862e81a0be5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:19:17.973476 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.973433 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e73e1c71-d9da-4c54-94e7-9862e81a0be5-tls-certs" pod "e73e1c71-d9da-4c54-94e7-9862e81a0be5" (UID: "e73e1c71-d9da-4c54-94e7-9862e81a0be5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:19:17.973540 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:17.973472 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73e1c71-d9da-4c54-94e7-9862e81a0be5-kube-api-access-whzv9" pod "e73e1c71-d9da-4c54-94e7-9862e81a0be5" (UID: "e73e1c71-d9da-4c54-94e7-9862e81a0be5"). InnerVolumeSpecName "kube-api-access-whzv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:19:18.073504 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:18.073440 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whzv9\" (UniqueName: \"kubernetes.io/projected/e73e1c71-d9da-4c54-94e7-9862e81a0be5-kube-api-access-whzv9\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:19:18.073504 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:18.073464 4861 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73e1c71-d9da-4c54-94e7-9862e81a0be5-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:19:18.073504 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:18.073475 4861 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73e1c71-d9da-4c54-94e7-9862e81a0be5-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:19:18.703833 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:18.703803 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" Oct 30 17:19:18.703833 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:18.703820 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c" event={"ID":"e73e1c71-d9da-4c54-94e7-9862e81a0be5","Type":"ContainerDied","Data":"8ffc138d3ea1637da58163986a93dd29e2b351131affe9e880834b1eb7b00654"} Oct 30 17:19:18.704390 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:18.703875 4861 scope.go:122] "RemoveContainer" containerID="c8e4998cf936752cab1ae1f7680ab05d920d5f70ed85600200c0f465ce2b212f" Oct 30 17:19:18.762594 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:18.762571 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c"] Oct 30 17:19:18.782301 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:18.782264 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6acc78/test-model-6acc78-kserve-7c5d8c7d9f-pww5c"] Oct 30 17:19:19.810773 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:19.810714 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73e1c71-d9da-4c54-94e7-9862e81a0be5" path="/var/lib/kubelet/pods/e73e1c71-d9da-4c54-94e7-9862e81a0be5/volumes" Oct 30 17:19:20.293312 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:19:20.293279 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:19:22.914884 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:22.914853 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl"] Oct 30 17:19:22.915906 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:22.915880 4861 cpu_manager.go:414] "containerMap: removing container" podUID="e73e1c71-d9da-4c54-94e7-9862e81a0be5" containerName="main" Oct 30 17:19:22.915906 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:22.915903 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73e1c71-d9da-4c54-94e7-9862e81a0be5" containerName="main" Oct 30 17:19:22.916221 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:22.916204 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73e1c71-d9da-4c54-94e7-9862e81a0be5" containerName="main" Oct 30 17:19:22.920195 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:22.920175 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:22.942131 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:22.942101 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl"] Oct 30 17:19:22.949022 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:22.948985 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0b09d3-serving-cert\"" Oct 30 17:19:22.962976 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:22.959037 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-maas-api-tls\") pod \"maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:22.962976 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:22.959188 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-metrics-certs\") pod \"maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:22.962976 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:22.959220 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef02b08b-f178-425d-86ce-0e72bdcb63a2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:22.962976 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:22.959262 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v94fr\" (UniqueName: \"kubernetes.io/projected/ef02b08b-f178-425d-86ce-0e72bdcb63a2-kube-api-access-v94fr\") pod \"maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:23.061360 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:23.061332 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-maas-api-tls\") pod \"maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:23.061532 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:23.061440 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-metrics-certs\") pod \"maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:23.061532 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:23.061460 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef02b08b-f178-425d-86ce-0e72bdcb63a2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:23.061532 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:19:23.061476 4861 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-serving-cert: secret "maas-api-e2e-inf-b-0b09d3-serving-cert" not found Oct 30 17:19:23.061532 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:23.061502 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v94fr\" (UniqueName: \"kubernetes.io/projected/ef02b08b-f178-425d-86ce-0e72bdcb63a2-kube-api-access-v94fr\") pod \"maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:23.061532 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:19:23.061530 4861 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-maas-api-tls podName:ef02b08b-f178-425d-86ce-0e72bdcb63a2 nodeName:}" failed. No retries permitted until 2026-10-30 17:19:23.561515504 +0000 UTC m=+3393.904189544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-maas-api-tls") pod "maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" (UID: "ef02b08b-f178-425d-86ce-0e72bdcb63a2") : secret "maas-api-e2e-inf-b-0b09d3-serving-cert" not found Oct 30 17:19:23.062237 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:23.062214 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef02b08b-f178-425d-86ce-0e72bdcb63a2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:23.064271 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:23.064247 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-metrics-certs\") pod \"maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:23.086706 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:23.086682 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v94fr\" (UniqueName: \"kubernetes.io/projected/ef02b08b-f178-425d-86ce-0e72bdcb63a2-kube-api-access-v94fr\") pod \"maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:23.569715 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:23.569680 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-maas-api-tls\") pod \"maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:23.572485 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:23.572457 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-maas-api-tls\") pod \"maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:23.835876 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:23.835803 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:24.150568 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:19:24.150534 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef02b08b_f178_425d_86ce_0e72bdcb63a2.slice/crio-efc7d118dd3c9d049d866ac3413cf07d18efa843b395224684ef3e9a18f7c4f9 WatchSource:0}: Error finding container efc7d118dd3c9d049d866ac3413cf07d18efa843b395224684ef3e9a18f7c4f9: Status 404 returned error can't find the container with id efc7d118dd3c9d049d866ac3413cf07d18efa843b395224684ef3e9a18f7c4f9 Oct 30 17:19:24.152076 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:24.152052 4861 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:19:24.152801 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:24.152772 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl"] Oct 30 17:19:24.779535 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:24.779498 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" event={"ID":"ef02b08b-f178-425d-86ce-0e72bdcb63a2","Type":"ContainerStarted","Data":"fb96e2f8d229af0498d28fe9c2bd7ed89f49a0932d5fb0af01098ded811cde00"} Oct 30 17:19:24.779741 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:24.779562 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" event={"ID":"ef02b08b-f178-425d-86ce-0e72bdcb63a2","Type":"ContainerStarted","Data":"efc7d118dd3c9d049d866ac3413cf07d18efa843b395224684ef3e9a18f7c4f9"} Oct 30 17:19:24.779741 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:24.779593 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:24.826144 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:24.826096 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" podStartSLOduration=2.383143727 podStartE2EDuration="2.826081868s" totalImagesPullingTime="442.938141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:24.825801626 +0000 UTC m=+3395.168475664" watchObservedRunningTime="2026-10-30 17:19:24.826081868 +0000 UTC m=+3395.168755905" Oct 30 17:19:30.797430 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:19:30.797397 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:19:50.292572 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:19:50.292525 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:20:20.298420 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:20:20.298376 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:20:26.016263 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:26.016223 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb"] Oct 30 17:20:31.016923 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:31.016819 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" podUID="1f65a199-0838-4aa9-85d4-fb082370d14c" containerName="payload-pre-processing" containerID="cri-o://f852edf94e0df1be64bb72d4eab661aa4a48dc0b3ced32280103e8f4a8abc2dd" gracePeriod=25 Oct 30 17:20:33.395189 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:33.395146 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" podUID="1f65a199-0838-4aa9-85d4-fb082370d14c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:20:43.394566 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:43.394522 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" podUID="1f65a199-0838-4aa9-85d4-fb082370d14c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:20:50.293405 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:20:50.293367 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:20:51.329311 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:51.329280 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" Oct 30 17:20:51.409698 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:51.409666 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f65a199-0838-4aa9-85d4-fb082370d14c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f65a199-0838-4aa9-85d4-fb082370d14c-plugins-config-volume\") pod \"1f65a199-0838-4aa9-85d4-fb082370d14c\" (UID: \"1f65a199-0838-4aa9-85d4-fb082370d14c\") " Oct 30 17:20:51.409943 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:51.409924 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f65a199-0838-4aa9-85d4-fb082370d14c-kube-api-access-cs8ls\" (UniqueName: \"kubernetes.io/projected/1f65a199-0838-4aa9-85d4-fb082370d14c-kube-api-access-cs8ls\") pod \"1f65a199-0838-4aa9-85d4-fb082370d14c\" (UID: \"1f65a199-0838-4aa9-85d4-fb082370d14c\") " Oct 30 17:20:51.410141 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:51.410113 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f65a199-0838-4aa9-85d4-fb082370d14c-plugins-config-volume" pod "1f65a199-0838-4aa9-85d4-fb082370d14c" (UID: "1f65a199-0838-4aa9-85d4-fb082370d14c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:20:51.410696 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:51.410622 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f65a199-0838-4aa9-85d4-fb082370d14c-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:20:51.412289 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:51.412265 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f65a199-0838-4aa9-85d4-fb082370d14c-kube-api-access-cs8ls" pod "1f65a199-0838-4aa9-85d4-fb082370d14c" (UID: "1f65a199-0838-4aa9-85d4-fb082370d14c"). InnerVolumeSpecName "kube-api-access-cs8ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:51.511927 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:51.511899 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs8ls\" (UniqueName: \"kubernetes.io/projected/1f65a199-0838-4aa9-85d4-fb082370d14c-kube-api-access-cs8ls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:20:51.897207 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:51.897175 4861 generic.go:356] "Generic (PLEG): container finished" podID="1f65a199-0838-4aa9-85d4-fb082370d14c" containerID="f852edf94e0df1be64bb72d4eab661aa4a48dc0b3ced32280103e8f4a8abc2dd" exitCode=0 Oct 30 17:20:51.897207 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:51.897207 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" Oct 30 17:20:51.897415 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:51.897259 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" event={"ID":"1f65a199-0838-4aa9-85d4-fb082370d14c","Type":"ContainerDied","Data":"f852edf94e0df1be64bb72d4eab661aa4a48dc0b3ced32280103e8f4a8abc2dd"} Oct 30 17:20:51.897415 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:51.897315 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb" event={"ID":"1f65a199-0838-4aa9-85d4-fb082370d14c","Type":"ContainerDied","Data":"49311fc336c11b18950f17c0f081bebcc4240ecd1a6e667d33a75c179e751073"} Oct 30 17:20:51.897415 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:51.897340 4861 scope.go:122] "RemoveContainer" containerID="f852edf94e0df1be64bb72d4eab661aa4a48dc0b3ced32280103e8f4a8abc2dd" Oct 30 17:20:51.909671 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:51.909648 4861 scope.go:122] "RemoveContainer" containerID="f852edf94e0df1be64bb72d4eab661aa4a48dc0b3ced32280103e8f4a8abc2dd" Oct 30 17:20:51.910018 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:20:51.909996 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f852edf94e0df1be64bb72d4eab661aa4a48dc0b3ced32280103e8f4a8abc2dd\": container with ID starting with f852edf94e0df1be64bb72d4eab661aa4a48dc0b3ced32280103e8f4a8abc2dd not found: ID does not exist" containerID="f852edf94e0df1be64bb72d4eab661aa4a48dc0b3ced32280103e8f4a8abc2dd" Oct 30 17:20:51.910116 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:51.910024 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f852edf94e0df1be64bb72d4eab661aa4a48dc0b3ced32280103e8f4a8abc2dd"} err="failed to get container status \"f852edf94e0df1be64bb72d4eab661aa4a48dc0b3ced32280103e8f4a8abc2dd\": rpc error: code = NotFound desc = could not find container \"f852edf94e0df1be64bb72d4eab661aa4a48dc0b3ced32280103e8f4a8abc2dd\": container with ID starting with f852edf94e0df1be64bb72d4eab661aa4a48dc0b3ced32280103e8f4a8abc2dd not found: ID does not exist" Oct 30 17:20:51.955913 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:51.955889 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb"] Oct 30 17:20:51.973970 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:51.973938 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24acb2-c8bb9f5f-87pqb"] Oct 30 17:20:53.810361 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:20:53.810325 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f65a199-0838-4aa9-85d4-fb082370d14c" path="/var/lib/kubelet/pods/1f65a199-0838-4aa9-85d4-fb082370d14c/volumes" Oct 30 17:21:19.284454 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:19.284419 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52"] Oct 30 17:21:19.284955 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:19.284660 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" podUID="8ed523f1-369d-4559-a4be-74633818bc17" containerName="maas-api" containerID="cri-o://3dbe1f69681558ccdd47418ec8d69a1e6cd7f54174669e602e3236ff152d5695" gracePeriod=30 Oct 30 17:21:19.774755 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:19.771908 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:21:19.865747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:19.865545 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ed523f1-369d-4559-a4be-74633818bc17-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ed523f1-369d-4559-a4be-74633818bc17-maas-api-tls\") pod \"8ed523f1-369d-4559-a4be-74633818bc17\" (UID: \"8ed523f1-369d-4559-a4be-74633818bc17\") " Oct 30 17:21:19.865747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:19.865679 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ed523f1-369d-4559-a4be-74633818bc17-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ed523f1-369d-4559-a4be-74633818bc17-metrics-certs\") pod \"8ed523f1-369d-4559-a4be-74633818bc17\" (UID: \"8ed523f1-369d-4559-a4be-74633818bc17\") " Oct 30 17:21:19.866026 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:19.865996 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed523f1-369d-4559-a4be-74633818bc17-kube-api-access-mg4dl\" (UniqueName: \"kubernetes.io/projected/8ed523f1-369d-4559-a4be-74633818bc17-kube-api-access-mg4dl\") pod \"8ed523f1-369d-4559-a4be-74633818bc17\" (UID: \"8ed523f1-369d-4559-a4be-74633818bc17\") " Oct 30 17:21:19.866341 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:19.866094 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed523f1-369d-4559-a4be-74633818bc17-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed523f1-369d-4559-a4be-74633818bc17-openshift-service-ca\") pod \"8ed523f1-369d-4559-a4be-74633818bc17\" (UID: \"8ed523f1-369d-4559-a4be-74633818bc17\") " Oct 30 17:21:19.886786 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:19.877659 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed523f1-369d-4559-a4be-74633818bc17-openshift-service-ca" pod "8ed523f1-369d-4559-a4be-74633818bc17" (UID: "8ed523f1-369d-4559-a4be-74633818bc17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:19.920196 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:19.920090 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed523f1-369d-4559-a4be-74633818bc17-maas-api-tls" pod "8ed523f1-369d-4559-a4be-74633818bc17" (UID: "8ed523f1-369d-4559-a4be-74633818bc17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:19.923944 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:19.922466 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed523f1-369d-4559-a4be-74633818bc17-kube-api-access-mg4dl" pod "8ed523f1-369d-4559-a4be-74633818bc17" (UID: "8ed523f1-369d-4559-a4be-74633818bc17"). InnerVolumeSpecName "kube-api-access-mg4dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:19.937549 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:19.933028 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed523f1-369d-4559-a4be-74633818bc17-metrics-certs" pod "8ed523f1-369d-4559-a4be-74633818bc17" (UID: "8ed523f1-369d-4559-a4be-74633818bc17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:19.969336 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:19.969294 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ed523f1-369d-4559-a4be-74633818bc17-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:19.969336 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:19.969327 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ed523f1-369d-4559-a4be-74633818bc17-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:19.969532 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:19.969343 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg4dl\" (UniqueName: \"kubernetes.io/projected/8ed523f1-369d-4559-a4be-74633818bc17-kube-api-access-mg4dl\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:19.969532 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:19.969357 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ed523f1-369d-4559-a4be-74633818bc17-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:20.145447 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:20.145406 4861 generic.go:356] "Generic (PLEG): container finished" podID="8ed523f1-369d-4559-a4be-74633818bc17" containerID="3dbe1f69681558ccdd47418ec8d69a1e6cd7f54174669e602e3236ff152d5695" exitCode=0 Oct 30 17:21:20.145665 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:20.145455 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" event={"ID":"8ed523f1-369d-4559-a4be-74633818bc17","Type":"ContainerDied","Data":"3dbe1f69681558ccdd47418ec8d69a1e6cd7f54174669e602e3236ff152d5695"} Oct 30 17:21:20.145665 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:20.145466 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" Oct 30 17:21:20.150366 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:20.150072 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52" event={"ID":"8ed523f1-369d-4559-a4be-74633818bc17","Type":"ContainerDied","Data":"cdc66c1ed69ab23e0299ceb5fa7d15326622aadecc635f3e19a1833ef56cde8a"} Oct 30 17:21:20.150366 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:20.150106 4861 scope.go:122] "RemoveContainer" containerID="3dbe1f69681558ccdd47418ec8d69a1e6cd7f54174669e602e3236ff152d5695" Oct 30 17:21:20.163716 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:20.163696 4861 scope.go:122] "RemoveContainer" containerID="3dbe1f69681558ccdd47418ec8d69a1e6cd7f54174669e602e3236ff152d5695" Oct 30 17:21:20.164077 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:21:20.164057 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dbe1f69681558ccdd47418ec8d69a1e6cd7f54174669e602e3236ff152d5695\": container with ID starting with 3dbe1f69681558ccdd47418ec8d69a1e6cd7f54174669e602e3236ff152d5695 not found: ID does not exist" containerID="3dbe1f69681558ccdd47418ec8d69a1e6cd7f54174669e602e3236ff152d5695" Oct 30 17:21:20.164139 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:20.164085 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbe1f69681558ccdd47418ec8d69a1e6cd7f54174669e602e3236ff152d5695"} err="failed to get container status \"3dbe1f69681558ccdd47418ec8d69a1e6cd7f54174669e602e3236ff152d5695\": rpc error: code = NotFound desc = could not find container \"3dbe1f69681558ccdd47418ec8d69a1e6cd7f54174669e602e3236ff152d5695\": container with ID starting with 3dbe1f69681558ccdd47418ec8d69a1e6cd7f54174669e602e3236ff152d5695 not found: ID does not exist" Oct 30 17:21:20.283314 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:20.283282 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52"] Oct 30 17:21:20.301894 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:21:20.301855 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:21:20.321460 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:20.321429 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-24acb2-65f7bcd6c5-bxp52"] Oct 30 17:21:20.373923 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:20.373889 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v"] Oct 30 17:21:20.374206 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:20.374176 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" podUID="26f54ab0-924f-4d9d-aa69-69e9015b1bd5" containerName="istio-proxy" containerID="cri-o://0c07378670c3f0e4b55498a44b69ac5885300af96c2d215acd4b24ee89b66e86" gracePeriod=30 Oct 30 17:21:21.810279 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:21.810244 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ed523f1-369d-4559-a4be-74633818bc17" path="/var/lib/kubelet/pods/8ed523f1-369d-4559-a4be-74633818bc17/volumes" Oct 30 17:21:25.679087 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.679057 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:21:25.793088 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.793053 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-envoy\") pod \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " Oct 30 17:21:25.793600 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.793575 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-data\") pod \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " Oct 30 17:21:25.793845 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.793825 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-workload-certs\") pod \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " Oct 30 17:21:25.793963 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.793933 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-data" pod "26f54ab0-924f-4d9d-aa69-69e9015b1bd5" (UID: "26f54ab0-924f-4d9d-aa69-69e9015b1bd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:25.794132 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.794111 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-token\" (UniqueName: \"kubernetes.io/projected/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-token\") pod \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " Oct 30 17:21:25.794306 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.794290 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istiod-ca-cert\") pod \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " Oct 30 17:21:25.794629 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.794609 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-credential-socket\") pod \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " Oct 30 17:21:25.794891 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.794871 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-podinfo\") pod \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " Oct 30 17:21:25.795035 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.795018 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-workload-socket\") pod \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " Oct 30 17:21:25.795223 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.795203 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-kube-api-access-2q44k\" (UniqueName: \"kubernetes.io/projected/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-kube-api-access-2q44k\") pod \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\" (UID: \"26f54ab0-924f-4d9d-aa69-69e9015b1bd5\") " Oct 30 17:21:25.795518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.794902 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-workload-certs" pod "26f54ab0-924f-4d9d-aa69-69e9015b1bd5" (UID: "26f54ab0-924f-4d9d-aa69-69e9015b1bd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:25.795788 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.795763 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-workload-socket" pod "26f54ab0-924f-4d9d-aa69-69e9015b1bd5" (UID: "26f54ab0-924f-4d9d-aa69-69e9015b1bd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:25.796111 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.796086 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istiod-ca-cert" pod "26f54ab0-924f-4d9d-aa69-69e9015b1bd5" (UID: "26f54ab0-924f-4d9d-aa69-69e9015b1bd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:25.796494 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.796472 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:25.796617 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.796603 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:25.796742 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.796709 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:25.796850 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.796837 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:25.796980 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.796707 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-credential-socket" pod "26f54ab0-924f-4d9d-aa69-69e9015b1bd5" (UID: "26f54ab0-924f-4d9d-aa69-69e9015b1bd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:25.798941 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.798908 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-kube-api-access-2q44k" pod "26f54ab0-924f-4d9d-aa69-69e9015b1bd5" (UID: "26f54ab0-924f-4d9d-aa69-69e9015b1bd5"). InnerVolumeSpecName "kube-api-access-2q44k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:25.799047 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.798952 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-token" pod "26f54ab0-924f-4d9d-aa69-69e9015b1bd5" (UID: "26f54ab0-924f-4d9d-aa69-69e9015b1bd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:25.808815 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.808788 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-envoy" pod "26f54ab0-924f-4d9d-aa69-69e9015b1bd5" (UID: "26f54ab0-924f-4d9d-aa69-69e9015b1bd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:25.818909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.818881 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-podinfo" pod "26f54ab0-924f-4d9d-aa69-69e9015b1bd5" (UID: "26f54ab0-924f-4d9d-aa69-69e9015b1bd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:25.900005 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.899901 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:25.900005 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.899939 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:25.900005 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.899953 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:25.900005 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.899963 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q44k\" (UniqueName: \"kubernetes.io/projected/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-kube-api-access-2q44k\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:25.900005 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:25.899975 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26f54ab0-924f-4d9d-aa69-69e9015b1bd5-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:26.209167 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:26.209079 4861 generic.go:356] "Generic (PLEG): container finished" podID="26f54ab0-924f-4d9d-aa69-69e9015b1bd5" containerID="0c07378670c3f0e4b55498a44b69ac5885300af96c2d215acd4b24ee89b66e86" exitCode=0 Oct 30 17:21:26.209167 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:26.209131 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" Oct 30 17:21:26.209369 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:26.209177 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" event={"ID":"26f54ab0-924f-4d9d-aa69-69e9015b1bd5","Type":"ContainerDied","Data":"0c07378670c3f0e4b55498a44b69ac5885300af96c2d215acd4b24ee89b66e86"} Oct 30 17:21:26.209369 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:26.209233 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v" event={"ID":"26f54ab0-924f-4d9d-aa69-69e9015b1bd5","Type":"ContainerDied","Data":"f1dc08ab59e37b32b5a9a806643ceb425c482920a84970639ff9e56f23dbcb40"} Oct 30 17:21:26.209369 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:26.209258 4861 scope.go:122] "RemoveContainer" containerID="0c07378670c3f0e4b55498a44b69ac5885300af96c2d215acd4b24ee89b66e86" Oct 30 17:21:26.221201 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:26.221183 4861 scope.go:122] "RemoveContainer" containerID="0c07378670c3f0e4b55498a44b69ac5885300af96c2d215acd4b24ee89b66e86" Oct 30 17:21:26.221470 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:21:26.221448 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c07378670c3f0e4b55498a44b69ac5885300af96c2d215acd4b24ee89b66e86\": container with ID starting with 0c07378670c3f0e4b55498a44b69ac5885300af96c2d215acd4b24ee89b66e86 not found: ID does not exist" containerID="0c07378670c3f0e4b55498a44b69ac5885300af96c2d215acd4b24ee89b66e86" Oct 30 17:21:26.221524 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:26.221477 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c07378670c3f0e4b55498a44b69ac5885300af96c2d215acd4b24ee89b66e86"} err="failed to get container status \"0c07378670c3f0e4b55498a44b69ac5885300af96c2d215acd4b24ee89b66e86\": rpc error: code = NotFound desc = could not find container \"0c07378670c3f0e4b55498a44b69ac5885300af96c2d215acd4b24ee89b66e86\": container with ID starting with 0c07378670c3f0e4b55498a44b69ac5885300af96c2d215acd4b24ee89b66e86 not found: ID does not exist" Oct 30 17:21:26.269060 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:26.269030 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v"] Oct 30 17:21:26.273400 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:26.273380 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a77b48-openshift-default-f5b8c9bcc-2hw5v"] Oct 30 17:21:27.823312 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:27.823273 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f54ab0-924f-4d9d-aa69-69e9015b1bd5" path="/var/lib/kubelet/pods/26f54ab0-924f-4d9d-aa69-69e9015b1bd5/volumes" Oct 30 17:21:31.474421 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:31.474389 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl"] Oct 30 17:21:31.474895 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:31.474587 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" podUID="9e67b1b9-0721-4914-85ed-6abf64720a33" containerName="istio-proxy" containerID="cri-o://70137b579187c78386389b70f9cebdd26ec7de8cd220f414531a79332306d5b3" gracePeriod=30 Oct 30 17:21:34.779832 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:34.779797 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l"] Oct 30 17:21:34.780347 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:34.780009 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" podUID="20bdae97-71b8-4376-85b4-157138986bb2" containerName="istio-proxy" containerID="cri-o://14317835bc3ae9b0e2fd12eb31bd18903ecc3fa4ace244d35069fd281cdf1d99" gracePeriod=30 Oct 30 17:21:35.310651 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:35.310618 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv"] Oct 30 17:21:35.310928 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:35.310882 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" podUID="7adb920d-0538-46bb-8c96-3d5f68902e50" containerName="istio-proxy" containerID="cri-o://f9f009d6735de488925d6eb3767cc4919921a289738c60b1633effe2cb972bb4" gracePeriod=30 Oct 30 17:21:35.315968 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:35.315943 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv"] Oct 30 17:21:35.316136 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:35.316117 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" podUID="cd5c0df9-6522-46f6-8222-117b7ad17bea" containerName="istio-proxy" containerID="cri-o://d4e4b0cfcc4b59396039dd445a7c8d4e67cf3e3feef5f0dd61c540b7946063ce" gracePeriod=30 Oct 30 17:21:35.960738 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:35.960682 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr"] Oct 30 17:21:35.961217 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:35.960944 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" podUID="854e3a2c-0257-4f16-96ba-e86cc2ba282d" containerName="istio-proxy" containerID="cri-o://9ad0a003cd360964e6b65e6b9da3d895a4ee1cf3279657002acd9c0400694271" gracePeriod=30 Oct 30 17:21:36.777204 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.777180 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:21:36.798949 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.798919 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e67b1b9-0721-4914-85ed-6abf64720a33-kube-api-access-6b7h8\" (UniqueName: \"kubernetes.io/projected/9e67b1b9-0721-4914-85ed-6abf64720a33-kube-api-access-6b7h8\") pod \"9e67b1b9-0721-4914-85ed-6abf64720a33\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " Oct 30 17:21:36.799123 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.799024 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e67b1b9-0721-4914-85ed-6abf64720a33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e67b1b9-0721-4914-85ed-6abf64720a33-istiod-ca-cert\") pod \"9e67b1b9-0721-4914-85ed-6abf64720a33\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " Oct 30 17:21:36.799123 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.799088 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-workload-certs\") pod \"9e67b1b9-0721-4914-85ed-6abf64720a33\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " Oct 30 17:21:36.799265 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.799161 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-token\" (UniqueName: \"kubernetes.io/projected/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-token\") pod \"9e67b1b9-0721-4914-85ed-6abf64720a33\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " Oct 30 17:21:36.799328 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.799308 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-data\") pod \"9e67b1b9-0721-4914-85ed-6abf64720a33\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " Oct 30 17:21:36.799387 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.799337 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-workload-socket\") pod \"9e67b1b9-0721-4914-85ed-6abf64720a33\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " Oct 30 17:21:36.799446 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.799399 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-envoy\") pod \"9e67b1b9-0721-4914-85ed-6abf64720a33\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " Oct 30 17:21:36.799512 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.799476 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e67b1b9-0721-4914-85ed-6abf64720a33-istiod-ca-cert" pod "9e67b1b9-0721-4914-85ed-6abf64720a33" (UID: "9e67b1b9-0721-4914-85ed-6abf64720a33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:36.799822 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.799782 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-workload-socket" pod "9e67b1b9-0721-4914-85ed-6abf64720a33" (UID: "9e67b1b9-0721-4914-85ed-6abf64720a33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:36.800013 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.799889 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-workload-certs" pod "9e67b1b9-0721-4914-85ed-6abf64720a33" (UID: "9e67b1b9-0721-4914-85ed-6abf64720a33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:36.800228 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.800067 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-data" pod "9e67b1b9-0721-4914-85ed-6abf64720a33" (UID: "9e67b1b9-0721-4914-85ed-6abf64720a33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:36.800342 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.800287 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-podinfo\") pod \"9e67b1b9-0721-4914-85ed-6abf64720a33\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " Oct 30 17:21:36.800485 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.800461 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-credential-socket\") pod \"9e67b1b9-0721-4914-85ed-6abf64720a33\" (UID: \"9e67b1b9-0721-4914-85ed-6abf64720a33\") " Oct 30 17:21:36.800757 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.800715 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-credential-socket" pod "9e67b1b9-0721-4914-85ed-6abf64720a33" (UID: "9e67b1b9-0721-4914-85ed-6abf64720a33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:36.801251 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.801222 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e67b1b9-0721-4914-85ed-6abf64720a33-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:36.801353 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.801254 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:36.801353 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.801267 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:36.801353 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.801279 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:36.801353 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.801292 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:36.801963 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.801942 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-token" pod "9e67b1b9-0721-4914-85ed-6abf64720a33" (UID: "9e67b1b9-0721-4914-85ed-6abf64720a33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:36.802167 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.802142 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e67b1b9-0721-4914-85ed-6abf64720a33-kube-api-access-6b7h8" pod "9e67b1b9-0721-4914-85ed-6abf64720a33" (UID: "9e67b1b9-0721-4914-85ed-6abf64720a33"). InnerVolumeSpecName "kube-api-access-6b7h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:36.802275 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.802258 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-envoy" pod "9e67b1b9-0721-4914-85ed-6abf64720a33" (UID: "9e67b1b9-0721-4914-85ed-6abf64720a33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:36.802663 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.802647 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-podinfo" pod "9e67b1b9-0721-4914-85ed-6abf64720a33" (UID: "9e67b1b9-0721-4914-85ed-6abf64720a33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:36.903115 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.903070 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:36.903115 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.903105 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:36.903115 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.903120 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e67b1b9-0721-4914-85ed-6abf64720a33-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:36.903425 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:36.903135 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b7h8\" (UniqueName: \"kubernetes.io/projected/9e67b1b9-0721-4914-85ed-6abf64720a33-kube-api-access-6b7h8\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:37.316361 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:37.316328 4861 generic.go:356] "Generic (PLEG): container finished" podID="9e67b1b9-0721-4914-85ed-6abf64720a33" containerID="70137b579187c78386389b70f9cebdd26ec7de8cd220f414531a79332306d5b3" exitCode=0 Oct 30 17:21:37.316866 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:37.316400 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" Oct 30 17:21:37.316866 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:37.316448 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" event={"ID":"9e67b1b9-0721-4914-85ed-6abf64720a33","Type":"ContainerDied","Data":"70137b579187c78386389b70f9cebdd26ec7de8cd220f414531a79332306d5b3"} Oct 30 17:21:37.316866 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:37.316501 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl" event={"ID":"9e67b1b9-0721-4914-85ed-6abf64720a33","Type":"ContainerDied","Data":"1a3a678ce527f9d754feae713a0128d23a8f29cbee43247b0159c6b44f5b3ac1"} Oct 30 17:21:37.316866 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:37.316533 4861 scope.go:122] "RemoveContainer" containerID="70137b579187c78386389b70f9cebdd26ec7de8cd220f414531a79332306d5b3" Oct 30 17:21:37.342354 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:37.342324 4861 scope.go:122] "RemoveContainer" containerID="70137b579187c78386389b70f9cebdd26ec7de8cd220f414531a79332306d5b3" Oct 30 17:21:37.342674 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:21:37.342646 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70137b579187c78386389b70f9cebdd26ec7de8cd220f414531a79332306d5b3\": container with ID starting with 70137b579187c78386389b70f9cebdd26ec7de8cd220f414531a79332306d5b3 not found: ID does not exist" containerID="70137b579187c78386389b70f9cebdd26ec7de8cd220f414531a79332306d5b3" Oct 30 17:21:37.342786 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:37.342683 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70137b579187c78386389b70f9cebdd26ec7de8cd220f414531a79332306d5b3"} err="failed to get container status \"70137b579187c78386389b70f9cebdd26ec7de8cd220f414531a79332306d5b3\": rpc error: code = NotFound desc = could not find container \"70137b579187c78386389b70f9cebdd26ec7de8cd220f414531a79332306d5b3\": container with ID starting with 70137b579187c78386389b70f9cebdd26ec7de8cd220f414531a79332306d5b3 not found: ID does not exist" Oct 30 17:21:37.391207 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:37.391178 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl"] Oct 30 17:21:37.407490 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:37.407462 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-3ca339-openshift-default-7d9d77487-m6xbl"] Oct 30 17:21:37.812294 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:37.812251 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e67b1b9-0721-4914-85ed-6abf64720a33" path="/var/lib/kubelet/pods/9e67b1b9-0721-4914-85ed-6abf64720a33/volumes" Oct 30 17:21:38.889135 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:38.889105 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2"] Oct 30 17:21:40.231862 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.231839 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:21:40.249799 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.249768 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-istio-data\") pod \"20bdae97-71b8-4376-85b4-157138986bb2\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " Oct 30 17:21:40.249942 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.249882 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-credential-socket\") pod \"20bdae97-71b8-4376-85b4-157138986bb2\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " Oct 30 17:21:40.249942 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.249905 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-istio-envoy\") pod \"20bdae97-71b8-4376-85b4-157138986bb2\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " Oct 30 17:21:40.249942 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.249930 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20bdae97-71b8-4376-85b4-157138986bb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20bdae97-71b8-4376-85b4-157138986bb2-istio-podinfo\") pod \"20bdae97-71b8-4376-85b4-157138986bb2\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " Oct 30 17:21:40.250128 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.249973 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20bdae97-71b8-4376-85b4-157138986bb2-kube-api-access-fg4w4\" (UniqueName: \"kubernetes.io/projected/20bdae97-71b8-4376-85b4-157138986bb2-kube-api-access-fg4w4\") pod \"20bdae97-71b8-4376-85b4-157138986bb2\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " Oct 30 17:21:40.250128 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.249982 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-istio-data" pod "20bdae97-71b8-4376-85b4-157138986bb2" (UID: "20bdae97-71b8-4376-85b4-157138986bb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:40.250128 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.249990 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20bdae97-71b8-4376-85b4-157138986bb2-istio-token\" (UniqueName: \"kubernetes.io/projected/20bdae97-71b8-4376-85b4-157138986bb2-istio-token\") pod \"20bdae97-71b8-4376-85b4-157138986bb2\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " Oct 30 17:21:40.250302 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.250161 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-workload-certs\") pod \"20bdae97-71b8-4376-85b4-157138986bb2\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " Oct 30 17:21:40.250302 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.250199 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20bdae97-71b8-4376-85b4-157138986bb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20bdae97-71b8-4376-85b4-157138986bb2-istiod-ca-cert\") pod \"20bdae97-71b8-4376-85b4-157138986bb2\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " Oct 30 17:21:40.250302 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.250208 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-credential-socket" pod "20bdae97-71b8-4376-85b4-157138986bb2" (UID: "20bdae97-71b8-4376-85b4-157138986bb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:40.250302 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.250234 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-workload-socket\") pod \"20bdae97-71b8-4376-85b4-157138986bb2\" (UID: \"20bdae97-71b8-4376-85b4-157138986bb2\") " Oct 30 17:21:40.250633 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.250443 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-workload-certs" pod "20bdae97-71b8-4376-85b4-157138986bb2" (UID: "20bdae97-71b8-4376-85b4-157138986bb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:40.250633 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.250530 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20bdae97-71b8-4376-85b4-157138986bb2-istiod-ca-cert" pod "20bdae97-71b8-4376-85b4-157138986bb2" (UID: "20bdae97-71b8-4376-85b4-157138986bb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:40.250775 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.250667 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-workload-socket" pod "20bdae97-71b8-4376-85b4-157138986bb2" (UID: "20bdae97-71b8-4376-85b4-157138986bb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:40.251298 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.251273 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.251365 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.251308 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.251365 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.251326 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.251365 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.251341 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20bdae97-71b8-4376-85b4-157138986bb2-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.251365 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.251356 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.252549 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.252529 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20bdae97-71b8-4376-85b4-157138986bb2-istio-podinfo" pod "20bdae97-71b8-4376-85b4-157138986bb2" (UID: "20bdae97-71b8-4376-85b4-157138986bb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:40.252662 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.252618 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20bdae97-71b8-4376-85b4-157138986bb2-istio-token" pod "20bdae97-71b8-4376-85b4-157138986bb2" (UID: "20bdae97-71b8-4376-85b4-157138986bb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:40.253051 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.253032 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-istio-envoy" pod "20bdae97-71b8-4376-85b4-157138986bb2" (UID: "20bdae97-71b8-4376-85b4-157138986bb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:40.253115 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.253074 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20bdae97-71b8-4376-85b4-157138986bb2-kube-api-access-fg4w4" pod "20bdae97-71b8-4376-85b4-157138986bb2" (UID: "20bdae97-71b8-4376-85b4-157138986bb2"). InnerVolumeSpecName "kube-api-access-fg4w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:40.353808 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.353772 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20bdae97-71b8-4376-85b4-157138986bb2-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.353961 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.353813 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20bdae97-71b8-4376-85b4-157138986bb2-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.353961 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.353829 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg4w4\" (UniqueName: \"kubernetes.io/projected/20bdae97-71b8-4376-85b4-157138986bb2-kube-api-access-fg4w4\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.353961 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.353845 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20bdae97-71b8-4376-85b4-157138986bb2-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.355416 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.355384 4861 generic.go:356] "Generic (PLEG): container finished" podID="20bdae97-71b8-4376-85b4-157138986bb2" containerID="14317835bc3ae9b0e2fd12eb31bd18903ecc3fa4ace244d35069fd281cdf1d99" exitCode=0 Oct 30 17:21:40.355572 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.355463 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" Oct 30 17:21:40.355572 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.355469 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" event={"ID":"20bdae97-71b8-4376-85b4-157138986bb2","Type":"ContainerDied","Data":"14317835bc3ae9b0e2fd12eb31bd18903ecc3fa4ace244d35069fd281cdf1d99"} Oct 30 17:21:40.355572 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.355548 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l" event={"ID":"20bdae97-71b8-4376-85b4-157138986bb2","Type":"ContainerDied","Data":"e29b48292a530e100fbca37db02a74c00d5d87b11df3bcad7f5477b7fec23a08"} Oct 30 17:21:40.355766 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.355580 4861 scope.go:122] "RemoveContainer" containerID="14317835bc3ae9b0e2fd12eb31bd18903ecc3fa4ace244d35069fd281cdf1d99" Oct 30 17:21:40.438838 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.438815 4861 scope.go:122] "RemoveContainer" containerID="14317835bc3ae9b0e2fd12eb31bd18903ecc3fa4ace244d35069fd281cdf1d99" Oct 30 17:21:40.439217 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:21:40.439191 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14317835bc3ae9b0e2fd12eb31bd18903ecc3fa4ace244d35069fd281cdf1d99\": container with ID starting with 14317835bc3ae9b0e2fd12eb31bd18903ecc3fa4ace244d35069fd281cdf1d99 not found: ID does not exist" containerID="14317835bc3ae9b0e2fd12eb31bd18903ecc3fa4ace244d35069fd281cdf1d99" Oct 30 17:21:40.439298 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.439226 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14317835bc3ae9b0e2fd12eb31bd18903ecc3fa4ace244d35069fd281cdf1d99"} err="failed to get container status \"14317835bc3ae9b0e2fd12eb31bd18903ecc3fa4ace244d35069fd281cdf1d99\": rpc error: code = NotFound desc = could not find container \"14317835bc3ae9b0e2fd12eb31bd18903ecc3fa4ace244d35069fd281cdf1d99\": container with ID starting with 14317835bc3ae9b0e2fd12eb31bd18903ecc3fa4ace244d35069fd281cdf1d99 not found: ID does not exist" Oct 30 17:21:40.487005 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.486952 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l"] Oct 30 17:21:40.500336 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.500290 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-13c495-openshift-default-7f4f449fcc-c946l"] Oct 30 17:21:40.542954 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.542798 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b"] Oct 30 17:21:40.543171 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.543111 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" podUID="24e3b70b-25df-4464-be5e-d5f77e1db8fd" containerName="istio-proxy" containerID="cri-o://bf39ab140b9a86374f6e5dde7da50e0fe5a64db959fb5e25ff4bd7b56ff95a68" gracePeriod=30 Oct 30 17:21:40.609891 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.609863 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:21:40.657702 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.657672 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-podinfo\") pod \"cd5c0df9-6522-46f6-8222-117b7ad17bea\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " Oct 30 17:21:40.657873 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.657797 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-workload-certs\") pod \"cd5c0df9-6522-46f6-8222-117b7ad17bea\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " Oct 30 17:21:40.657873 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.657857 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-credential-socket\") pod \"cd5c0df9-6522-46f6-8222-117b7ad17bea\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " Oct 30 17:21:40.658012 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.657893 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd5c0df9-6522-46f6-8222-117b7ad17bea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd5c0df9-6522-46f6-8222-117b7ad17bea-istiod-ca-cert\") pod \"cd5c0df9-6522-46f6-8222-117b7ad17bea\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " Oct 30 17:21:40.658012 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.657938 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-envoy\") pod \"cd5c0df9-6522-46f6-8222-117b7ad17bea\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " Oct 30 17:21:40.658106 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.658075 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-token\" (UniqueName: \"kubernetes.io/projected/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-token\") pod \"cd5c0df9-6522-46f6-8222-117b7ad17bea\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " Oct 30 17:21:40.658165 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.658133 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-data\") pod \"cd5c0df9-6522-46f6-8222-117b7ad17bea\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " Oct 30 17:21:40.658891 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.658280 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-workload-socket\") pod \"cd5c0df9-6522-46f6-8222-117b7ad17bea\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " Oct 30 17:21:40.658891 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.658328 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd5c0df9-6522-46f6-8222-117b7ad17bea-kube-api-access-pfj5k\" (UniqueName: \"kubernetes.io/projected/cd5c0df9-6522-46f6-8222-117b7ad17bea-kube-api-access-pfj5k\") pod \"cd5c0df9-6522-46f6-8222-117b7ad17bea\" (UID: \"cd5c0df9-6522-46f6-8222-117b7ad17bea\") " Oct 30 17:21:40.658891 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.658382 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-credential-socket" pod "cd5c0df9-6522-46f6-8222-117b7ad17bea" (UID: "cd5c0df9-6522-46f6-8222-117b7ad17bea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:40.658891 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.658404 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-workload-certs" pod "cd5c0df9-6522-46f6-8222-117b7ad17bea" (UID: "cd5c0df9-6522-46f6-8222-117b7ad17bea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:40.658891 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.658465 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd5c0df9-6522-46f6-8222-117b7ad17bea-istiod-ca-cert" pod "cd5c0df9-6522-46f6-8222-117b7ad17bea" (UID: "cd5c0df9-6522-46f6-8222-117b7ad17bea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:40.658891 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.658790 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-data" pod "cd5c0df9-6522-46f6-8222-117b7ad17bea" (UID: "cd5c0df9-6522-46f6-8222-117b7ad17bea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:40.660009 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.659079 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-workload-socket" pod "cd5c0df9-6522-46f6-8222-117b7ad17bea" (UID: "cd5c0df9-6522-46f6-8222-117b7ad17bea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:40.660009 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.659949 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.660009 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.659967 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.660009 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.659976 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.660009 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.659986 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd5c0df9-6522-46f6-8222-117b7ad17bea-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.660009 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.660001 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.661251 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.661228 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-token" pod "cd5c0df9-6522-46f6-8222-117b7ad17bea" (UID: "cd5c0df9-6522-46f6-8222-117b7ad17bea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:40.662148 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.662126 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:21:40.664803 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.664779 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-podinfo" pod "cd5c0df9-6522-46f6-8222-117b7ad17bea" (UID: "cd5c0df9-6522-46f6-8222-117b7ad17bea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:40.668870 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.668848 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-envoy" pod "cd5c0df9-6522-46f6-8222-117b7ad17bea" (UID: "cd5c0df9-6522-46f6-8222-117b7ad17bea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:40.669323 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.669300 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd5c0df9-6522-46f6-8222-117b7ad17bea-kube-api-access-pfj5k" pod "cd5c0df9-6522-46f6-8222-117b7ad17bea" (UID: "cd5c0df9-6522-46f6-8222-117b7ad17bea"). InnerVolumeSpecName "kube-api-access-pfj5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:40.761068 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.761040 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-podinfo\") pod \"7adb920d-0538-46bb-8c96-3d5f68902e50\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " Oct 30 17:21:40.761303 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.761167 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-credential-socket\") pod \"7adb920d-0538-46bb-8c96-3d5f68902e50\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " Oct 30 17:21:40.761303 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.761224 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-envoy\") pod \"7adb920d-0538-46bb-8c96-3d5f68902e50\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " Oct 30 17:21:40.761303 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.761266 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7adb920d-0538-46bb-8c96-3d5f68902e50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adb920d-0538-46bb-8c96-3d5f68902e50-istiod-ca-cert\") pod \"7adb920d-0538-46bb-8c96-3d5f68902e50\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " Oct 30 17:21:40.761519 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.761308 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7adb920d-0538-46bb-8c96-3d5f68902e50-kube-api-access-h6pv5\" (UniqueName: \"kubernetes.io/projected/7adb920d-0538-46bb-8c96-3d5f68902e50-kube-api-access-h6pv5\") pod \"7adb920d-0538-46bb-8c96-3d5f68902e50\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " Oct 30 17:21:40.761519 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.761332 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-workload-certs\") pod \"7adb920d-0538-46bb-8c96-3d5f68902e50\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " Oct 30 17:21:40.761519 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.761420 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-token\" (UniqueName: \"kubernetes.io/projected/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-token\") pod \"7adb920d-0538-46bb-8c96-3d5f68902e50\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " Oct 30 17:21:40.761519 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.761489 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-workload-socket\") pod \"7adb920d-0538-46bb-8c96-3d5f68902e50\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " Oct 30 17:21:40.761519 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.761495 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-credential-socket" pod "7adb920d-0538-46bb-8c96-3d5f68902e50" (UID: "7adb920d-0538-46bb-8c96-3d5f68902e50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:40.761848 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.761648 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-data\") pod \"7adb920d-0538-46bb-8c96-3d5f68902e50\" (UID: \"7adb920d-0538-46bb-8c96-3d5f68902e50\") " Oct 30 17:21:40.761848 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.761693 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-workload-socket" pod "7adb920d-0538-46bb-8c96-3d5f68902e50" (UID: "7adb920d-0538-46bb-8c96-3d5f68902e50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:40.761848 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.761700 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7adb920d-0538-46bb-8c96-3d5f68902e50-istiod-ca-cert" pod "7adb920d-0538-46bb-8c96-3d5f68902e50" (UID: "7adb920d-0538-46bb-8c96-3d5f68902e50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:40.762039 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.761904 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-workload-certs" pod "7adb920d-0538-46bb-8c96-3d5f68902e50" (UID: "7adb920d-0538-46bb-8c96-3d5f68902e50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:40.762198 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.762175 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-data" pod "7adb920d-0538-46bb-8c96-3d5f68902e50" (UID: "7adb920d-0538-46bb-8c96-3d5f68902e50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:40.762886 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.762857 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.762997 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.762890 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adb920d-0538-46bb-8c96-3d5f68902e50-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.762997 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.762905 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.762997 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.762918 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.762997 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.762930 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.762997 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.762942 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd5c0df9-6522-46f6-8222-117b7ad17bea-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.762997 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.762954 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.762997 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.762967 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfj5k\" (UniqueName: \"kubernetes.io/projected/cd5c0df9-6522-46f6-8222-117b7ad17bea-kube-api-access-pfj5k\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.762997 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.762990 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.763902 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.763875 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-envoy" pod "7adb920d-0538-46bb-8c96-3d5f68902e50" (UID: "7adb920d-0538-46bb-8c96-3d5f68902e50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:40.764155 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.764120 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-token" pod "7adb920d-0538-46bb-8c96-3d5f68902e50" (UID: "7adb920d-0538-46bb-8c96-3d5f68902e50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:40.764255 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.764197 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-podinfo" pod "7adb920d-0538-46bb-8c96-3d5f68902e50" (UID: "7adb920d-0538-46bb-8c96-3d5f68902e50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:40.764255 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.764208 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7adb920d-0538-46bb-8c96-3d5f68902e50-kube-api-access-h6pv5" pod "7adb920d-0538-46bb-8c96-3d5f68902e50" (UID: "7adb920d-0538-46bb-8c96-3d5f68902e50"). InnerVolumeSpecName "kube-api-access-h6pv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:40.869426 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.869385 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.869426 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.869424 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6pv5\" (UniqueName: \"kubernetes.io/projected/7adb920d-0538-46bb-8c96-3d5f68902e50-kube-api-access-h6pv5\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.869737 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.869441 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:40.869737 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:40.869452 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adb920d-0538-46bb-8c96-3d5f68902e50-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.272794 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.272766 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:21:41.368982 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.368891 4861 generic.go:356] "Generic (PLEG): container finished" podID="7adb920d-0538-46bb-8c96-3d5f68902e50" containerID="f9f009d6735de488925d6eb3767cc4919921a289738c60b1633effe2cb972bb4" exitCode=0 Oct 30 17:21:41.368982 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.368959 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" Oct 30 17:21:41.369240 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.368981 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" event={"ID":"7adb920d-0538-46bb-8c96-3d5f68902e50","Type":"ContainerDied","Data":"f9f009d6735de488925d6eb3767cc4919921a289738c60b1633effe2cb972bb4"} Oct 30 17:21:41.369240 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.369059 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv" event={"ID":"7adb920d-0538-46bb-8c96-3d5f68902e50","Type":"ContainerDied","Data":"671728be9011423659da0f24856205ef687d71a2f3c5ca163da75150cb2cfee0"} Oct 30 17:21:41.369240 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.369094 4861 scope.go:122] "RemoveContainer" containerID="f9f009d6735de488925d6eb3767cc4919921a289738c60b1633effe2cb972bb4" Oct 30 17:21:41.370797 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.370759 4861 generic.go:356] "Generic (PLEG): container finished" podID="854e3a2c-0257-4f16-96ba-e86cc2ba282d" containerID="9ad0a003cd360964e6b65e6b9da3d895a4ee1cf3279657002acd9c0400694271" exitCode=0 Oct 30 17:21:41.370797 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.370791 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" Oct 30 17:21:41.371019 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.370850 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" event={"ID":"854e3a2c-0257-4f16-96ba-e86cc2ba282d","Type":"ContainerDied","Data":"9ad0a003cd360964e6b65e6b9da3d895a4ee1cf3279657002acd9c0400694271"} Oct 30 17:21:41.371019 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.370905 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr" event={"ID":"854e3a2c-0257-4f16-96ba-e86cc2ba282d","Type":"ContainerDied","Data":"20c14f342e77425f62bf27285449aa758b877b0df8e4a4934aa08108bdf32a2b"} Oct 30 17:21:41.372563 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.372536 4861 generic.go:356] "Generic (PLEG): container finished" podID="cd5c0df9-6522-46f6-8222-117b7ad17bea" containerID="d4e4b0cfcc4b59396039dd445a7c8d4e67cf3e3feef5f0dd61c540b7946063ce" exitCode=0 Oct 30 17:21:41.372695 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.372606 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" event={"ID":"cd5c0df9-6522-46f6-8222-117b7ad17bea","Type":"ContainerDied","Data":"d4e4b0cfcc4b59396039dd445a7c8d4e67cf3e3feef5f0dd61c540b7946063ce"} Oct 30 17:21:41.372695 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.372645 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" event={"ID":"cd5c0df9-6522-46f6-8222-117b7ad17bea","Type":"ContainerDied","Data":"7f4bf6b8561943c4655f3ea2c7bd67e7f3fa59ca4de8babda7305d4ea914e4b3"} Oct 30 17:21:41.372825 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.372710 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv" Oct 30 17:21:41.376782 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.376709 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-workload-socket\") pod \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " Oct 30 17:21:41.376929 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.376854 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-credential-socket\") pod \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " Oct 30 17:21:41.377038 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.376930 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-data\") pod \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " Oct 30 17:21:41.377038 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.377024 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854e3a2c-0257-4f16-96ba-e86cc2ba282d-kube-api-access-lvz7c\" (UniqueName: \"kubernetes.io/projected/854e3a2c-0257-4f16-96ba-e86cc2ba282d-kube-api-access-lvz7c\") pod \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " Oct 30 17:21:41.377209 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.377131 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-workload-socket" pod "854e3a2c-0257-4f16-96ba-e86cc2ba282d" (UID: "854e3a2c-0257-4f16-96ba-e86cc2ba282d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:41.377209 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.377144 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-podinfo\") pod \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " Oct 30 17:21:41.377209 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.377176 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-workload-certs\") pod \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " Oct 30 17:21:41.377422 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.377295 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-credential-socket" pod "854e3a2c-0257-4f16-96ba-e86cc2ba282d" (UID: "854e3a2c-0257-4f16-96ba-e86cc2ba282d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:41.377529 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.377430 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-workload-certs" pod "854e3a2c-0257-4f16-96ba-e86cc2ba282d" (UID: "854e3a2c-0257-4f16-96ba-e86cc2ba282d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:41.377529 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.377490 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-data" pod "854e3a2c-0257-4f16-96ba-e86cc2ba282d" (UID: "854e3a2c-0257-4f16-96ba-e86cc2ba282d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:41.377663 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.377534 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-envoy\") pod \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " Oct 30 17:21:41.377774 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.377753 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-token\" (UniqueName: \"kubernetes.io/projected/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-token\") pod \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " Oct 30 17:21:41.377883 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.377864 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istiod-ca-cert\") pod \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\" (UID: \"854e3a2c-0257-4f16-96ba-e86cc2ba282d\") " Oct 30 17:21:41.378768 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.378695 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.378873 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.378771 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.378873 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.378788 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.378873 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.378803 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.379288 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.379126 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istiod-ca-cert" pod "854e3a2c-0257-4f16-96ba-e86cc2ba282d" (UID: "854e3a2c-0257-4f16-96ba-e86cc2ba282d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:41.381314 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.379959 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-podinfo" pod "854e3a2c-0257-4f16-96ba-e86cc2ba282d" (UID: "854e3a2c-0257-4f16-96ba-e86cc2ba282d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:41.381314 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.381177 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-token" pod "854e3a2c-0257-4f16-96ba-e86cc2ba282d" (UID: "854e3a2c-0257-4f16-96ba-e86cc2ba282d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:41.382939 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.382861 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-envoy" pod "854e3a2c-0257-4f16-96ba-e86cc2ba282d" (UID: "854e3a2c-0257-4f16-96ba-e86cc2ba282d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:41.383426 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.383398 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854e3a2c-0257-4f16-96ba-e86cc2ba282d-kube-api-access-lvz7c" pod "854e3a2c-0257-4f16-96ba-e86cc2ba282d" (UID: "854e3a2c-0257-4f16-96ba-e86cc2ba282d"). InnerVolumeSpecName "kube-api-access-lvz7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:41.402766 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.402709 4861 scope.go:122] "RemoveContainer" containerID="f9f009d6735de488925d6eb3767cc4919921a289738c60b1633effe2cb972bb4" Oct 30 17:21:41.404288 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:21:41.404142 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f009d6735de488925d6eb3767cc4919921a289738c60b1633effe2cb972bb4\": container with ID starting with f9f009d6735de488925d6eb3767cc4919921a289738c60b1633effe2cb972bb4 not found: ID does not exist" containerID="f9f009d6735de488925d6eb3767cc4919921a289738c60b1633effe2cb972bb4" Oct 30 17:21:41.404288 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.404183 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f009d6735de488925d6eb3767cc4919921a289738c60b1633effe2cb972bb4"} err="failed to get container status \"f9f009d6735de488925d6eb3767cc4919921a289738c60b1633effe2cb972bb4\": rpc error: code = NotFound desc = could not find container \"f9f009d6735de488925d6eb3767cc4919921a289738c60b1633effe2cb972bb4\": container with ID starting with f9f009d6735de488925d6eb3767cc4919921a289738c60b1633effe2cb972bb4 not found: ID does not exist" Oct 30 17:21:41.404288 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.404206 4861 scope.go:122] "RemoveContainer" containerID="9ad0a003cd360964e6b65e6b9da3d895a4ee1cf3279657002acd9c0400694271" Oct 30 17:21:41.417640 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.417618 4861 scope.go:122] "RemoveContainer" containerID="9ad0a003cd360964e6b65e6b9da3d895a4ee1cf3279657002acd9c0400694271" Oct 30 17:21:41.417930 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:21:41.417903 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad0a003cd360964e6b65e6b9da3d895a4ee1cf3279657002acd9c0400694271\": container with ID starting with 9ad0a003cd360964e6b65e6b9da3d895a4ee1cf3279657002acd9c0400694271 not found: ID does not exist" containerID="9ad0a003cd360964e6b65e6b9da3d895a4ee1cf3279657002acd9c0400694271" Oct 30 17:21:41.417994 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.417940 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad0a003cd360964e6b65e6b9da3d895a4ee1cf3279657002acd9c0400694271"} err="failed to get container status \"9ad0a003cd360964e6b65e6b9da3d895a4ee1cf3279657002acd9c0400694271\": rpc error: code = NotFound desc = could not find container \"9ad0a003cd360964e6b65e6b9da3d895a4ee1cf3279657002acd9c0400694271\": container with ID starting with 9ad0a003cd360964e6b65e6b9da3d895a4ee1cf3279657002acd9c0400694271 not found: ID does not exist" Oct 30 17:21:41.417994 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.417960 4861 scope.go:122] "RemoveContainer" containerID="d4e4b0cfcc4b59396039dd445a7c8d4e67cf3e3feef5f0dd61c540b7946063ce" Oct 30 17:21:41.429872 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.429781 4861 scope.go:122] "RemoveContainer" containerID="d4e4b0cfcc4b59396039dd445a7c8d4e67cf3e3feef5f0dd61c540b7946063ce" Oct 30 17:21:41.430122 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:21:41.430103 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e4b0cfcc4b59396039dd445a7c8d4e67cf3e3feef5f0dd61c540b7946063ce\": container with ID starting with d4e4b0cfcc4b59396039dd445a7c8d4e67cf3e3feef5f0dd61c540b7946063ce not found: ID does not exist" containerID="d4e4b0cfcc4b59396039dd445a7c8d4e67cf3e3feef5f0dd61c540b7946063ce" Oct 30 17:21:41.430200 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.430131 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e4b0cfcc4b59396039dd445a7c8d4e67cf3e3feef5f0dd61c540b7946063ce"} err="failed to get container status \"d4e4b0cfcc4b59396039dd445a7c8d4e67cf3e3feef5f0dd61c540b7946063ce\": rpc error: code = NotFound desc = could not find container \"d4e4b0cfcc4b59396039dd445a7c8d4e67cf3e3feef5f0dd61c540b7946063ce\": container with ID starting with d4e4b0cfcc4b59396039dd445a7c8d4e67cf3e3feef5f0dd61c540b7946063ce not found: ID does not exist" Oct 30 17:21:41.475304 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.475271 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv"] Oct 30 17:21:41.480759 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.480691 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.480759 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.480718 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.480957 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.480772 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvz7c\" (UniqueName: \"kubernetes.io/projected/854e3a2c-0257-4f16-96ba-e86cc2ba282d-kube-api-access-lvz7c\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.480957 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.480787 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.480957 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.480815 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/854e3a2c-0257-4f16-96ba-e86cc2ba282d-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:41.482494 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.482458 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-g2hzv"] Oct 30 17:21:41.527990 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.527935 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv"] Oct 30 17:21:41.532506 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.532476 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7854949c9-9s7vv"] Oct 30 17:21:41.723734 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.723683 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr"] Oct 30 17:21:41.727944 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.727917 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-18519c-openshift-default-7fb6c8ddff-76xtr"] Oct 30 17:21:41.813680 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.813651 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20bdae97-71b8-4376-85b4-157138986bb2" path="/var/lib/kubelet/pods/20bdae97-71b8-4376-85b4-157138986bb2/volumes" Oct 30 17:21:41.814365 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.814346 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7adb920d-0538-46bb-8c96-3d5f68902e50" path="/var/lib/kubelet/pods/7adb920d-0538-46bb-8c96-3d5f68902e50/volumes" Oct 30 17:21:41.815035 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.815018 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="854e3a2c-0257-4f16-96ba-e86cc2ba282d" path="/var/lib/kubelet/pods/854e3a2c-0257-4f16-96ba-e86cc2ba282d/volumes" Oct 30 17:21:41.816134 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:41.816113 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd5c0df9-6522-46f6-8222-117b7ad17bea" path="/var/lib/kubelet/pods/cd5c0df9-6522-46f6-8222-117b7ad17bea/volumes" Oct 30 17:21:43.889952 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:43.889913 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" podUID="d7d2da07-ef07-4524-8627-6e2c5a201e6e" containerName="payload-processing" containerID="cri-o://0127b6db41c010658a5d9bba91666853ef3fb5e0410dc056262d20e9c83498c5" gracePeriod=25 Oct 30 17:21:45.719019 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:45.718973 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk"] Oct 30 17:21:45.719506 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:45.719199 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" podUID="29ca4500-8d2b-4e41-8ae2-13f452ded9c3" containerName="maas-api" containerID="cri-o://9339b658b2039763aaa35c7cc7eba7c1e37c45aa853bf55bf20db54bedee743b" gracePeriod=30 Oct 30 17:21:45.905185 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:45.905159 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:21:46.081296 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.081260 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e3b70b-25df-4464-be5e-d5f77e1db8fd-kube-api-access-tdmkr\" (UniqueName: \"kubernetes.io/projected/24e3b70b-25df-4464-be5e-d5f77e1db8fd-kube-api-access-tdmkr\") pod \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " Oct 30 17:21:46.081552 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.081326 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-envoy\") pod \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " Oct 30 17:21:46.081552 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.081413 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-credential-socket\") pod \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " Oct 30 17:21:46.081552 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.081487 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-data\") pod \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " Oct 30 17:21:46.081552 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.081547 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-token\" (UniqueName: \"kubernetes.io/projected/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-token\") pod \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " Oct 30 17:21:46.081848 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.081672 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-workload-socket\") pod \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " Oct 30 17:21:46.081848 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.081800 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-workload-certs\") pod \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " Oct 30 17:21:46.081848 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.081840 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istiod-ca-cert\") pod \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " Oct 30 17:21:46.081963 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.081899 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-podinfo\") pod \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\" (UID: \"24e3b70b-25df-4464-be5e-d5f77e1db8fd\") " Oct 30 17:21:46.082776 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.082524 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-data" pod "24e3b70b-25df-4464-be5e-d5f77e1db8fd" (UID: "24e3b70b-25df-4464-be5e-d5f77e1db8fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:46.084983 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.083177 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-workload-socket" pod "24e3b70b-25df-4464-be5e-d5f77e1db8fd" (UID: "24e3b70b-25df-4464-be5e-d5f77e1db8fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:46.084983 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.083551 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-workload-certs" pod "24e3b70b-25df-4464-be5e-d5f77e1db8fd" (UID: "24e3b70b-25df-4464-be5e-d5f77e1db8fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:46.084983 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.084936 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istiod-ca-cert" pod "24e3b70b-25df-4464-be5e-d5f77e1db8fd" (UID: "24e3b70b-25df-4464-be5e-d5f77e1db8fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:46.085332 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.085288 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-credential-socket" pod "24e3b70b-25df-4464-be5e-d5f77e1db8fd" (UID: "24e3b70b-25df-4464-be5e-d5f77e1db8fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:46.086021 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.085993 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:46.086261 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.086026 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:46.086261 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.086042 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:46.086261 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.086056 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:46.086261 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.086071 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:46.090107 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.090086 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-envoy" pod "24e3b70b-25df-4464-be5e-d5f77e1db8fd" (UID: "24e3b70b-25df-4464-be5e-d5f77e1db8fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:21:46.091211 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.091189 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-token" pod "24e3b70b-25df-4464-be5e-d5f77e1db8fd" (UID: "24e3b70b-25df-4464-be5e-d5f77e1db8fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:46.092267 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.091960 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:21:46.093027 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.092959 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e3b70b-25df-4464-be5e-d5f77e1db8fd-kube-api-access-tdmkr" pod "24e3b70b-25df-4464-be5e-d5f77e1db8fd" (UID: "24e3b70b-25df-4464-be5e-d5f77e1db8fd"). InnerVolumeSpecName "kube-api-access-tdmkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:46.093027 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.092990 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-podinfo" pod "24e3b70b-25df-4464-be5e-d5f77e1db8fd" (UID: "24e3b70b-25df-4464-be5e-d5f77e1db8fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:21:46.193800 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.193290 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-kube-api-access-6724k\" (UniqueName: \"kubernetes.io/projected/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-kube-api-access-6724k\") pod \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\" (UID: \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\") " Oct 30 17:21:46.193800 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.193382 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-metrics-certs\") pod \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\" (UID: \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\") " Oct 30 17:21:46.193800 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.193463 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-maas-api-tls\") pod \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\" (UID: \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\") " Oct 30 17:21:46.193800 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.193670 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-openshift-service-ca\") pod \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\" (UID: \"29ca4500-8d2b-4e41-8ae2-13f452ded9c3\") " Oct 30 17:21:46.198743 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.194442 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:46.198743 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.194471 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdmkr\" (UniqueName: \"kubernetes.io/projected/24e3b70b-25df-4464-be5e-d5f77e1db8fd-kube-api-access-tdmkr\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:46.198743 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.194487 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:46.198743 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.194501 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e3b70b-25df-4464-be5e-d5f77e1db8fd-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:46.210781 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.209092 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-metrics-certs" pod "29ca4500-8d2b-4e41-8ae2-13f452ded9c3" (UID: "29ca4500-8d2b-4e41-8ae2-13f452ded9c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:46.216680 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.211196 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-kube-api-access-6724k" pod "29ca4500-8d2b-4e41-8ae2-13f452ded9c3" (UID: "29ca4500-8d2b-4e41-8ae2-13f452ded9c3"). InnerVolumeSpecName "kube-api-access-6724k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:46.230127 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.230049 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-openshift-service-ca" pod "29ca4500-8d2b-4e41-8ae2-13f452ded9c3" (UID: "29ca4500-8d2b-4e41-8ae2-13f452ded9c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:46.230951 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.230918 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-maas-api-tls" pod "29ca4500-8d2b-4e41-8ae2-13f452ded9c3" (UID: "29ca4500-8d2b-4e41-8ae2-13f452ded9c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:46.298167 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.296772 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:46.298475 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.298451 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6724k\" (UniqueName: \"kubernetes.io/projected/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-kube-api-access-6724k\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:46.298600 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.298590 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:46.298688 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.298678 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29ca4500-8d2b-4e41-8ae2-13f452ded9c3-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:21:46.419012 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.418967 4861 generic.go:356] "Generic (PLEG): container finished" podID="24e3b70b-25df-4464-be5e-d5f77e1db8fd" containerID="bf39ab140b9a86374f6e5dde7da50e0fe5a64db959fb5e25ff4bd7b56ff95a68" exitCode=0 Oct 30 17:21:46.419353 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.419333 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" Oct 30 17:21:46.419740 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.419692 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" event={"ID":"24e3b70b-25df-4464-be5e-d5f77e1db8fd","Type":"ContainerDied","Data":"bf39ab140b9a86374f6e5dde7da50e0fe5a64db959fb5e25ff4bd7b56ff95a68"} Oct 30 17:21:46.419866 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.419778 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b" event={"ID":"24e3b70b-25df-4464-be5e-d5f77e1db8fd","Type":"ContainerDied","Data":"21958029dd37115b313f1a6bd659c3be5b03fa191c9d9341a4829cc8e82f6914"} Oct 30 17:21:46.419866 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.419815 4861 scope.go:122] "RemoveContainer" containerID="bf39ab140b9a86374f6e5dde7da50e0fe5a64db959fb5e25ff4bd7b56ff95a68" Oct 30 17:21:46.421608 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.421586 4861 generic.go:356] "Generic (PLEG): container finished" podID="29ca4500-8d2b-4e41-8ae2-13f452ded9c3" containerID="9339b658b2039763aaa35c7cc7eba7c1e37c45aa853bf55bf20db54bedee743b" exitCode=0 Oct 30 17:21:46.421696 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.421663 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" event={"ID":"29ca4500-8d2b-4e41-8ae2-13f452ded9c3","Type":"ContainerDied","Data":"9339b658b2039763aaa35c7cc7eba7c1e37c45aa853bf55bf20db54bedee743b"} Oct 30 17:21:46.421813 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.421703 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" event={"ID":"29ca4500-8d2b-4e41-8ae2-13f452ded9c3","Type":"ContainerDied","Data":"e1a9363e673d774e2d4902781b5f9c9b21fa78601fb3c70816973365cfdf8bb9"} Oct 30 17:21:46.421897 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.421814 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk" Oct 30 17:21:46.433629 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.433614 4861 scope.go:122] "RemoveContainer" containerID="bf39ab140b9a86374f6e5dde7da50e0fe5a64db959fb5e25ff4bd7b56ff95a68" Oct 30 17:21:46.433935 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:21:46.433913 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf39ab140b9a86374f6e5dde7da50e0fe5a64db959fb5e25ff4bd7b56ff95a68\": container with ID starting with bf39ab140b9a86374f6e5dde7da50e0fe5a64db959fb5e25ff4bd7b56ff95a68 not found: ID does not exist" containerID="bf39ab140b9a86374f6e5dde7da50e0fe5a64db959fb5e25ff4bd7b56ff95a68" Oct 30 17:21:46.434022 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.433944 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf39ab140b9a86374f6e5dde7da50e0fe5a64db959fb5e25ff4bd7b56ff95a68"} err="failed to get container status \"bf39ab140b9a86374f6e5dde7da50e0fe5a64db959fb5e25ff4bd7b56ff95a68\": rpc error: code = NotFound desc = could not find container \"bf39ab140b9a86374f6e5dde7da50e0fe5a64db959fb5e25ff4bd7b56ff95a68\": container with ID starting with bf39ab140b9a86374f6e5dde7da50e0fe5a64db959fb5e25ff4bd7b56ff95a68 not found: ID does not exist" Oct 30 17:21:46.434022 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.433967 4861 scope.go:122] "RemoveContainer" containerID="9339b658b2039763aaa35c7cc7eba7c1e37c45aa853bf55bf20db54bedee743b" Oct 30 17:21:46.445747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.445702 4861 scope.go:122] "RemoveContainer" containerID="9339b658b2039763aaa35c7cc7eba7c1e37c45aa853bf55bf20db54bedee743b" Oct 30 17:21:46.446023 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:21:46.446004 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9339b658b2039763aaa35c7cc7eba7c1e37c45aa853bf55bf20db54bedee743b\": container with ID starting with 9339b658b2039763aaa35c7cc7eba7c1e37c45aa853bf55bf20db54bedee743b not found: ID does not exist" containerID="9339b658b2039763aaa35c7cc7eba7c1e37c45aa853bf55bf20db54bedee743b" Oct 30 17:21:46.446060 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.446031 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9339b658b2039763aaa35c7cc7eba7c1e37c45aa853bf55bf20db54bedee743b"} err="failed to get container status \"9339b658b2039763aaa35c7cc7eba7c1e37c45aa853bf55bf20db54bedee743b\": rpc error: code = NotFound desc = could not find container \"9339b658b2039763aaa35c7cc7eba7c1e37c45aa853bf55bf20db54bedee743b\": container with ID starting with 9339b658b2039763aaa35c7cc7eba7c1e37c45aa853bf55bf20db54bedee743b not found: ID does not exist" Oct 30 17:21:46.480169 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.480132 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b"] Oct 30 17:21:46.485120 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.485094 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-420ff3-openshift-default-7894d6cd95-mjx2b"] Oct 30 17:21:46.525516 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.525489 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk"] Oct 30 17:21:46.532258 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:46.532225 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d5b798-bbb58d75-4xqtk"] Oct 30 17:21:47.809901 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:47.809869 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24e3b70b-25df-4464-be5e-d5f77e1db8fd" path="/var/lib/kubelet/pods/24e3b70b-25df-4464-be5e-d5f77e1db8fd/volumes" Oct 30 17:21:47.810351 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:47.810336 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29ca4500-8d2b-4e41-8ae2-13f452ded9c3" path="/var/lib/kubelet/pods/29ca4500-8d2b-4e41-8ae2-13f452ded9c3/volumes" Oct 30 17:21:50.291988 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:21:50.291951 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:21:52.221086 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:21:52.221045 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" podUID="d7d2da07-ef07-4524-8627-6e2c5a201e6e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:22:02.220894 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:02.220809 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" podUID="d7d2da07-ef07-4524-8627-6e2c5a201e6e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:22:04.197899 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.197871 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" Oct 30 17:22:04.374827 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.374787 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d2da07-ef07-4524-8627-6e2c5a201e6e-kube-api-access-4zc45\" (UniqueName: \"kubernetes.io/projected/d7d2da07-ef07-4524-8627-6e2c5a201e6e-kube-api-access-4zc45\") pod \"d7d2da07-ef07-4524-8627-6e2c5a201e6e\" (UID: \"d7d2da07-ef07-4524-8627-6e2c5a201e6e\") " Oct 30 17:22:04.375034 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.374863 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d2da07-ef07-4524-8627-6e2c5a201e6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d2da07-ef07-4524-8627-6e2c5a201e6e-plugins-config-volume\") pod \"d7d2da07-ef07-4524-8627-6e2c5a201e6e\" (UID: \"d7d2da07-ef07-4524-8627-6e2c5a201e6e\") " Oct 30 17:22:04.375218 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.375198 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d2da07-ef07-4524-8627-6e2c5a201e6e-plugins-config-volume" pod "d7d2da07-ef07-4524-8627-6e2c5a201e6e" (UID: "d7d2da07-ef07-4524-8627-6e2c5a201e6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:04.375429 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.375409 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7d2da07-ef07-4524-8627-6e2c5a201e6e-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:22:04.377159 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.377132 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d2da07-ef07-4524-8627-6e2c5a201e6e-kube-api-access-4zc45" pod "d7d2da07-ef07-4524-8627-6e2c5a201e6e" (UID: "d7d2da07-ef07-4524-8627-6e2c5a201e6e"). InnerVolumeSpecName "kube-api-access-4zc45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:04.476775 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.476753 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zc45\" (UniqueName: \"kubernetes.io/projected/d7d2da07-ef07-4524-8627-6e2c5a201e6e-kube-api-access-4zc45\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:22:04.593159 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.593122 4861 generic.go:356] "Generic (PLEG): container finished" podID="d7d2da07-ef07-4524-8627-6e2c5a201e6e" containerID="0127b6db41c010658a5d9bba91666853ef3fb5e0410dc056262d20e9c83498c5" exitCode=0 Oct 30 17:22:04.593366 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.593168 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" Oct 30 17:22:04.593366 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.593203 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" event={"ID":"d7d2da07-ef07-4524-8627-6e2c5a201e6e","Type":"ContainerDied","Data":"0127b6db41c010658a5d9bba91666853ef3fb5e0410dc056262d20e9c83498c5"} Oct 30 17:22:04.593366 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.593292 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2" event={"ID":"d7d2da07-ef07-4524-8627-6e2c5a201e6e","Type":"ContainerDied","Data":"3c03831aebb72d2c6223c3058244159205d25d52f8fd3984161f7c6378516dca"} Oct 30 17:22:04.593366 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.593321 4861 scope.go:122] "RemoveContainer" containerID="0127b6db41c010658a5d9bba91666853ef3fb5e0410dc056262d20e9c83498c5" Oct 30 17:22:04.605548 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.605526 4861 scope.go:122] "RemoveContainer" containerID="0127b6db41c010658a5d9bba91666853ef3fb5e0410dc056262d20e9c83498c5" Oct 30 17:22:04.605875 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:22:04.605851 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0127b6db41c010658a5d9bba91666853ef3fb5e0410dc056262d20e9c83498c5\": container with ID starting with 0127b6db41c010658a5d9bba91666853ef3fb5e0410dc056262d20e9c83498c5 not found: ID does not exist" containerID="0127b6db41c010658a5d9bba91666853ef3fb5e0410dc056262d20e9c83498c5" Oct 30 17:22:04.605945 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.605883 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0127b6db41c010658a5d9bba91666853ef3fb5e0410dc056262d20e9c83498c5"} err="failed to get container status \"0127b6db41c010658a5d9bba91666853ef3fb5e0410dc056262d20e9c83498c5\": rpc error: code = NotFound desc = could not find container \"0127b6db41c010658a5d9bba91666853ef3fb5e0410dc056262d20e9c83498c5\": container with ID starting with 0127b6db41c010658a5d9bba91666853ef3fb5e0410dc056262d20e9c83498c5 not found: ID does not exist" Oct 30 17:22:04.653944 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.653919 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2"] Oct 30 17:22:04.660327 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.660303 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d5b798-6d9ffd74cd-dx7l2"] Oct 30 17:22:04.999909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:04.999873 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm"] Oct 30 17:22:05.810817 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:05.810774 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d2da07-ef07-4524-8627-6e2c5a201e6e" path="/var/lib/kubelet/pods/d7d2da07-ef07-4524-8627-6e2c5a201e6e/volumes" Oct 30 17:22:10.000815 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:10.000773 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" podUID="2810edfd-dbe1-4bd4-9060-cf8c9e054741" containerName="payload-processing" containerID="cri-o://86f493c314a1ce4f30d5737a73148c12e474579855e613a9bd66b4caecc211f3" gracePeriod=25 Oct 30 17:22:18.940130 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:18.940097 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" podUID="2810edfd-dbe1-4bd4-9060-cf8c9e054741" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:22:20.291335 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:22:20.291296 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:22:28.954750 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:28.952964 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" podUID="2810edfd-dbe1-4bd4-9060-cf8c9e054741" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:22:30.290110 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:30.290079 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" Oct 30 17:22:30.427156 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:30.427125 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2810edfd-dbe1-4bd4-9060-cf8c9e054741-kube-api-access-64lzr\" (UniqueName: \"kubernetes.io/projected/2810edfd-dbe1-4bd4-9060-cf8c9e054741-kube-api-access-64lzr\") pod \"2810edfd-dbe1-4bd4-9060-cf8c9e054741\" (UID: \"2810edfd-dbe1-4bd4-9060-cf8c9e054741\") " Oct 30 17:22:30.427331 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:30.427177 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2810edfd-dbe1-4bd4-9060-cf8c9e054741-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2810edfd-dbe1-4bd4-9060-cf8c9e054741-plugins-config-volume\") pod \"2810edfd-dbe1-4bd4-9060-cf8c9e054741\" (UID: \"2810edfd-dbe1-4bd4-9060-cf8c9e054741\") " Oct 30 17:22:30.427629 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:30.427600 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2810edfd-dbe1-4bd4-9060-cf8c9e054741-plugins-config-volume" pod "2810edfd-dbe1-4bd4-9060-cf8c9e054741" (UID: "2810edfd-dbe1-4bd4-9060-cf8c9e054741"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:30.427871 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:30.427856 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2810edfd-dbe1-4bd4-9060-cf8c9e054741-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:22:30.429430 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:30.429407 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2810edfd-dbe1-4bd4-9060-cf8c9e054741-kube-api-access-64lzr" pod "2810edfd-dbe1-4bd4-9060-cf8c9e054741" (UID: "2810edfd-dbe1-4bd4-9060-cf8c9e054741"). InnerVolumeSpecName "kube-api-access-64lzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:30.528800 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:30.528717 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64lzr\" (UniqueName: \"kubernetes.io/projected/2810edfd-dbe1-4bd4-9060-cf8c9e054741-kube-api-access-64lzr\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:22:30.814155 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:30.814079 4861 generic.go:356] "Generic (PLEG): container finished" podID="2810edfd-dbe1-4bd4-9060-cf8c9e054741" containerID="86f493c314a1ce4f30d5737a73148c12e474579855e613a9bd66b4caecc211f3" exitCode=0 Oct 30 17:22:30.814155 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:30.814131 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" Oct 30 17:22:30.814345 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:30.814180 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" event={"ID":"2810edfd-dbe1-4bd4-9060-cf8c9e054741","Type":"ContainerDied","Data":"86f493c314a1ce4f30d5737a73148c12e474579855e613a9bd66b4caecc211f3"} Oct 30 17:22:30.814345 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:30.814239 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm" event={"ID":"2810edfd-dbe1-4bd4-9060-cf8c9e054741","Type":"ContainerDied","Data":"56d37b39aadd4a151a490429ce8aa257cd357fcdbb0846d2b6f73a07f95662c7"} Oct 30 17:22:30.814345 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:30.814268 4861 scope.go:122] "RemoveContainer" containerID="86f493c314a1ce4f30d5737a73148c12e474579855e613a9bd66b4caecc211f3" Oct 30 17:22:30.826197 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:30.826176 4861 scope.go:122] "RemoveContainer" containerID="86f493c314a1ce4f30d5737a73148c12e474579855e613a9bd66b4caecc211f3" Oct 30 17:22:30.826489 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:22:30.826467 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f493c314a1ce4f30d5737a73148c12e474579855e613a9bd66b4caecc211f3\": container with ID starting with 86f493c314a1ce4f30d5737a73148c12e474579855e613a9bd66b4caecc211f3 not found: ID does not exist" containerID="86f493c314a1ce4f30d5737a73148c12e474579855e613a9bd66b4caecc211f3" Oct 30 17:22:30.826542 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:30.826495 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f493c314a1ce4f30d5737a73148c12e474579855e613a9bd66b4caecc211f3"} err="failed to get container status \"86f493c314a1ce4f30d5737a73148c12e474579855e613a9bd66b4caecc211f3\": rpc error: code = NotFound desc = could not find container \"86f493c314a1ce4f30d5737a73148c12e474579855e613a9bd66b4caecc211f3\": container with ID starting with 86f493c314a1ce4f30d5737a73148c12e474579855e613a9bd66b4caecc211f3 not found: ID does not exist" Oct 30 17:22:30.872987 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:30.872959 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm"] Oct 30 17:22:30.886431 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:30.886405 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-abda73-5c47cc6b-sxdrm"] Oct 30 17:22:31.810714 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:31.810675 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2810edfd-dbe1-4bd4-9060-cf8c9e054741" path="/var/lib/kubelet/pods/2810edfd-dbe1-4bd4-9060-cf8c9e054741/volumes" Oct 30 17:22:50.291930 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:22:50.291886 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:22:50.480290 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:50.480254 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 17:22:50.511886 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:22:50.511859 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 17:23:04.244055 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:04.244019 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn"] Oct 30 17:23:15.810689 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:15.810656 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj"] Oct 30 17:23:15.830587 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:15.830557 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5"] Oct 30 17:23:19.268270 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.268215 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" podUID="e1779c0b-52e6-4c97-b02c-116f12d11ff9" containerName="main" containerID="cri-o://ae94c9b930b2b2c124d4f5d83c22df25077915cb2f08677f97c09ba1a014be9a" gracePeriod=45 Oct 30 17:23:19.302357 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.302328 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" podUID="e1779c0b-52e6-4c97-b02c-116f12d11ff9" containerName="main" probeResult="failure" output="Get \"https://10.132.1.43:8000/ready\": dial tcp 10.132.1.43:8000: connect: connection refused" Oct 30 17:23:19.568464 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.568436 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:23:19.661942 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.661912 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1779c0b-52e6-4c97-b02c-116f12d11ff9-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1779c0b-52e6-4c97-b02c-116f12d11ff9-tls-certs\") pod \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " Oct 30 17:23:19.662130 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.661956 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-tmp-dir\") pod \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " Oct 30 17:23:19.662130 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.661976 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-dshm\") pod \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " Oct 30 17:23:19.662268 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.662221 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-tmp-dir" pod "e1779c0b-52e6-4c97-b02c-116f12d11ff9" (UID: "e1779c0b-52e6-4c97-b02c-116f12d11ff9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:19.662268 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.662248 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-home\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-home\") pod \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " Oct 30 17:23:19.662391 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.662303 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-model-cache\") pod \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " Oct 30 17:23:19.662523 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.662499 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-home" pod "e1779c0b-52e6-4c97-b02c-116f12d11ff9" (UID: "e1779c0b-52e6-4c97-b02c-116f12d11ff9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:19.662616 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.662543 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-model-cache" pod "e1779c0b-52e6-4c97-b02c-116f12d11ff9" (UID: "e1779c0b-52e6-4c97-b02c-116f12d11ff9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:19.662616 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.662577 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1779c0b-52e6-4c97-b02c-116f12d11ff9-kube-api-access-qcw46\" (UniqueName: \"kubernetes.io/projected/e1779c0b-52e6-4c97-b02c-116f12d11ff9-kube-api-access-qcw46\") pod \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\" (UID: \"e1779c0b-52e6-4c97-b02c-116f12d11ff9\") " Oct 30 17:23:19.663518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.663490 4861 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:19.663518 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.663521 4861 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:19.663716 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.663535 4861 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:19.664628 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.664605 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1779c0b-52e6-4c97-b02c-116f12d11ff9-tls-certs" pod "e1779c0b-52e6-4c97-b02c-116f12d11ff9" (UID: "e1779c0b-52e6-4c97-b02c-116f12d11ff9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:19.664777 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.664661 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-dshm" pod "e1779c0b-52e6-4c97-b02c-116f12d11ff9" (UID: "e1779c0b-52e6-4c97-b02c-116f12d11ff9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:19.665211 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.665188 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1779c0b-52e6-4c97-b02c-116f12d11ff9-kube-api-access-qcw46" pod "e1779c0b-52e6-4c97-b02c-116f12d11ff9" (UID: "e1779c0b-52e6-4c97-b02c-116f12d11ff9"). InnerVolumeSpecName "kube-api-access-qcw46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:19.765304 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.765276 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcw46\" (UniqueName: \"kubernetes.io/projected/e1779c0b-52e6-4c97-b02c-116f12d11ff9-kube-api-access-qcw46\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:19.765304 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.765304 4861 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1779c0b-52e6-4c97-b02c-116f12d11ff9-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:19.765455 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:19.765314 4861 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1779c0b-52e6-4c97-b02c-116f12d11ff9-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:20.296459 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:20.296394 4861 generic.go:356] "Generic (PLEG): container finished" podID="e1779c0b-52e6-4c97-b02c-116f12d11ff9" containerID="ae94c9b930b2b2c124d4f5d83c22df25077915cb2f08677f97c09ba1a014be9a" exitCode=0 Oct 30 17:23:20.297103 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:20.296464 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" event={"ID":"e1779c0b-52e6-4c97-b02c-116f12d11ff9","Type":"ContainerDied","Data":"ae94c9b930b2b2c124d4f5d83c22df25077915cb2f08677f97c09ba1a014be9a"} Oct 30 17:23:20.297258 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:20.297140 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" event={"ID":"e1779c0b-52e6-4c97-b02c-116f12d11ff9","Type":"ContainerDied","Data":"97f527aa5609675b0a6f3efa7a03bdf5b6f74d045ec9feb578767d37a7e9169c"} Oct 30 17:23:20.297258 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:20.297172 4861 scope.go:122] "RemoveContainer" containerID="ae94c9b930b2b2c124d4f5d83c22df25077915cb2f08677f97c09ba1a014be9a" Oct 30 17:23:20.297390 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:20.296597 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn" Oct 30 17:23:20.315312 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:20.315291 4861 scope.go:122] "RemoveContainer" containerID="ae94c9b930b2b2c124d4f5d83c22df25077915cb2f08677f97c09ba1a014be9a" Oct 30 17:23:20.315624 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:23:20.315603 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae94c9b930b2b2c124d4f5d83c22df25077915cb2f08677f97c09ba1a014be9a\": container with ID starting with ae94c9b930b2b2c124d4f5d83c22df25077915cb2f08677f97c09ba1a014be9a not found: ID does not exist" containerID="ae94c9b930b2b2c124d4f5d83c22df25077915cb2f08677f97c09ba1a014be9a" Oct 30 17:23:20.315713 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:20.315630 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae94c9b930b2b2c124d4f5d83c22df25077915cb2f08677f97c09ba1a014be9a"} err="failed to get container status \"ae94c9b930b2b2c124d4f5d83c22df25077915cb2f08677f97c09ba1a014be9a\": rpc error: code = NotFound desc = could not find container \"ae94c9b930b2b2c124d4f5d83c22df25077915cb2f08677f97c09ba1a014be9a\": container with ID starting with ae94c9b930b2b2c124d4f5d83c22df25077915cb2f08677f97c09ba1a014be9a not found: ID does not exist" Oct 30 17:23:20.337946 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:23:20.337912 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:23:20.358341 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:20.358318 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn"] Oct 30 17:23:20.371510 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:20.371484 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e15d1f/test-model-e15d1f-kserve-76554c59c5-j7lsn"] Oct 30 17:23:20.811265 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:20.811227 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" podUID="cff713aa-db97-4366-ae04-c2bb57261598" containerName="payload-processing" containerID="cri-o://8d93b706b96f780e2f683a349de87f3ca93b7916ef7c7d0732adbab88199a671" gracePeriod=25 Oct 30 17:23:20.831605 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:20.831570 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" podUID="26c8b7b8-b899-4303-870f-dce7e002fb42" containerName="payload-pre-processing" containerID="cri-o://b44ce697cbe329b4ec99d74e148212d8df50a00b4a8f7c93c528406d9e737dc6" gracePeriod=25 Oct 30 17:23:21.810026 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:21.809992 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1779c0b-52e6-4c97-b02c-116f12d11ff9" path="/var/lib/kubelet/pods/e1779c0b-52e6-4c97-b02c-116f12d11ff9/volumes" Oct 30 17:23:24.802207 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:24.802169 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" podUID="26c8b7b8-b899-4303-870f-dce7e002fb42" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:23:24.802591 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:24.802169 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" podUID="cff713aa-db97-4366-ae04-c2bb57261598" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:23:29.141299 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:29.141264 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt"] Oct 30 17:23:29.176538 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:29.176508 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc"] Oct 30 17:23:34.142239 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:34.142136 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" podUID="1ce50453-ccab-4825-8ca9-b15579c1d0b4" containerName="payload-processing" containerID="cri-o://2c26e348f0500d26b63327cc84a24fa6ac741c54f142196b671bc37317793858" gracePeriod=25 Oct 30 17:23:34.177490 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:34.177443 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" podUID="6013ebf2-30f0-42a9-a31f-e66795df2b56" containerName="payload-pre-processing" containerID="cri-o://0d434a8083452c5d5d095456075efecc78d7926662d0d95d205821c8ba01b6b5" gracePeriod=25 Oct 30 17:23:34.467259 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:34.467215 4861 generic.go:356] "Generic (PLEG): container finished" podID="1ce50453-ccab-4825-8ca9-b15579c1d0b4" containerID="2c26e348f0500d26b63327cc84a24fa6ac741c54f142196b671bc37317793858" exitCode=0 Oct 30 17:23:34.467396 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:34.467280 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" event={"ID":"1ce50453-ccab-4825-8ca9-b15579c1d0b4","Type":"ContainerDied","Data":"2c26e348f0500d26b63327cc84a24fa6ac741c54f142196b671bc37317793858"} Oct 30 17:23:34.467396 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:34.467356 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" event={"ID":"1ce50453-ccab-4825-8ca9-b15579c1d0b4","Type":"ContainerDied","Data":"7d5a8b0a8a11e379418ef3961eb3c4185ae6e978b7e5ce5a211c08c333c8ae85"} Oct 30 17:23:34.467396 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:34.467374 4861 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d5a8b0a8a11e379418ef3961eb3c4185ae6e978b7e5ce5a211c08c333c8ae85" Oct 30 17:23:34.484520 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:34.484497 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" Oct 30 17:23:34.571671 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:34.571628 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce50453-ccab-4825-8ca9-b15579c1d0b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce50453-ccab-4825-8ca9-b15579c1d0b4-plugins-config-volume\") pod \"1ce50453-ccab-4825-8ca9-b15579c1d0b4\" (UID: \"1ce50453-ccab-4825-8ca9-b15579c1d0b4\") " Oct 30 17:23:34.571889 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:34.571777 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce50453-ccab-4825-8ca9-b15579c1d0b4-kube-api-access-776c2\" (UniqueName: \"kubernetes.io/projected/1ce50453-ccab-4825-8ca9-b15579c1d0b4-kube-api-access-776c2\") pod \"1ce50453-ccab-4825-8ca9-b15579c1d0b4\" (UID: \"1ce50453-ccab-4825-8ca9-b15579c1d0b4\") " Oct 30 17:23:34.572143 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:34.572113 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce50453-ccab-4825-8ca9-b15579c1d0b4-plugins-config-volume" pod "1ce50453-ccab-4825-8ca9-b15579c1d0b4" (UID: "1ce50453-ccab-4825-8ca9-b15579c1d0b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:34.572566 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:34.572541 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ce50453-ccab-4825-8ca9-b15579c1d0b4-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:34.574198 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:34.574172 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce50453-ccab-4825-8ca9-b15579c1d0b4-kube-api-access-776c2" pod "1ce50453-ccab-4825-8ca9-b15579c1d0b4" (UID: "1ce50453-ccab-4825-8ca9-b15579c1d0b4"). InnerVolumeSpecName "kube-api-access-776c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:34.673678 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:34.673643 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-776c2\" (UniqueName: \"kubernetes.io/projected/1ce50453-ccab-4825-8ca9-b15579c1d0b4-kube-api-access-776c2\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:34.802798 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:34.802757 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" podUID="26c8b7b8-b899-4303-870f-dce7e002fb42" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:23:34.802798 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:34.802770 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" podUID="cff713aa-db97-4366-ae04-c2bb57261598" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:23:35.475871 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:35.475839 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt" Oct 30 17:23:35.534330 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:35.534300 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt"] Oct 30 17:23:35.547528 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:35.547500 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e15d1f-6bc8d9d788-5glkt"] Oct 30 17:23:35.811520 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:35.811450 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce50453-ccab-4825-8ca9-b15579c1d0b4" path="/var/lib/kubelet/pods/1ce50453-ccab-4825-8ca9-b15579c1d0b4/volumes" Oct 30 17:23:35.833950 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:35.833923 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" podUID="6013ebf2-30f0-42a9-a31f-e66795df2b56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:23:36.483386 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:36.483355 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz"] Oct 30 17:23:37.175366 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:37.175329 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8"] Oct 30 17:23:41.137191 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.137157 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" Oct 30 17:23:41.199886 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.199856 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cff713aa-db97-4366-ae04-c2bb57261598-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff713aa-db97-4366-ae04-c2bb57261598-plugins-config-volume\") pod \"cff713aa-db97-4366-ae04-c2bb57261598\" (UID: \"cff713aa-db97-4366-ae04-c2bb57261598\") " Oct 30 17:23:41.200056 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.200010 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff713aa-db97-4366-ae04-c2bb57261598-kube-api-access-z5k5s\" (UniqueName: \"kubernetes.io/projected/cff713aa-db97-4366-ae04-c2bb57261598-kube-api-access-z5k5s\") pod \"cff713aa-db97-4366-ae04-c2bb57261598\" (UID: \"cff713aa-db97-4366-ae04-c2bb57261598\") " Oct 30 17:23:41.200317 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.200289 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff713aa-db97-4366-ae04-c2bb57261598-plugins-config-volume" pod "cff713aa-db97-4366-ae04-c2bb57261598" (UID: "cff713aa-db97-4366-ae04-c2bb57261598"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:41.201065 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.201039 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff713aa-db97-4366-ae04-c2bb57261598-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:41.203867 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.203498 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" Oct 30 17:23:41.204149 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.204121 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff713aa-db97-4366-ae04-c2bb57261598-kube-api-access-z5k5s" pod "cff713aa-db97-4366-ae04-c2bb57261598" (UID: "cff713aa-db97-4366-ae04-c2bb57261598"). InnerVolumeSpecName "kube-api-access-z5k5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:41.301750 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.301708 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c8b7b8-b899-4303-870f-dce7e002fb42-kube-api-access-mkqd9\" (UniqueName: \"kubernetes.io/projected/26c8b7b8-b899-4303-870f-dce7e002fb42-kube-api-access-mkqd9\") pod \"26c8b7b8-b899-4303-870f-dce7e002fb42\" (UID: \"26c8b7b8-b899-4303-870f-dce7e002fb42\") " Oct 30 17:23:41.301903 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.301813 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c8b7b8-b899-4303-870f-dce7e002fb42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c8b7b8-b899-4303-870f-dce7e002fb42-plugins-config-volume\") pod \"26c8b7b8-b899-4303-870f-dce7e002fb42\" (UID: \"26c8b7b8-b899-4303-870f-dce7e002fb42\") " Oct 30 17:23:41.302178 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.302144 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c8b7b8-b899-4303-870f-dce7e002fb42-plugins-config-volume" pod "26c8b7b8-b899-4303-870f-dce7e002fb42" (UID: "26c8b7b8-b899-4303-870f-dce7e002fb42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:41.302445 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.302425 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5k5s\" (UniqueName: \"kubernetes.io/projected/cff713aa-db97-4366-ae04-c2bb57261598-kube-api-access-z5k5s\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:41.302513 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.302452 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26c8b7b8-b899-4303-870f-dce7e002fb42-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:41.303972 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.303952 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c8b7b8-b899-4303-870f-dce7e002fb42-kube-api-access-mkqd9" pod "26c8b7b8-b899-4303-870f-dce7e002fb42" (UID: "26c8b7b8-b899-4303-870f-dce7e002fb42"). InnerVolumeSpecName "kube-api-access-mkqd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:41.404327 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.404269 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkqd9\" (UniqueName: \"kubernetes.io/projected/26c8b7b8-b899-4303-870f-dce7e002fb42-kube-api-access-mkqd9\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:41.484094 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.484057 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" podUID="b1995227-e125-48f5-8bf8-ff46ae4fc16e" containerName="payload-processing" containerID="cri-o://8f7d664e5d4c3424f82afa24419d27a9efea402cd3c4b898ea5080a6a322422d" gracePeriod=25 Oct 30 17:23:41.549767 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.549738 4861 generic.go:356] "Generic (PLEG): container finished" podID="26c8b7b8-b899-4303-870f-dce7e002fb42" containerID="b44ce697cbe329b4ec99d74e148212d8df50a00b4a8f7c93c528406d9e737dc6" exitCode=0 Oct 30 17:23:41.549909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.549785 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" Oct 30 17:23:41.549909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.549818 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" event={"ID":"26c8b7b8-b899-4303-870f-dce7e002fb42","Type":"ContainerDied","Data":"b44ce697cbe329b4ec99d74e148212d8df50a00b4a8f7c93c528406d9e737dc6"} Oct 30 17:23:41.549909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.549877 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5" event={"ID":"26c8b7b8-b899-4303-870f-dce7e002fb42","Type":"ContainerDied","Data":"b0702e038efb2d272e2e6b3068e2adbc16ccddce57ffe306c44062d12b3c6977"} Oct 30 17:23:41.549909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.549905 4861 scope.go:122] "RemoveContainer" containerID="b44ce697cbe329b4ec99d74e148212d8df50a00b4a8f7c93c528406d9e737dc6" Oct 30 17:23:41.551553 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.551527 4861 generic.go:356] "Generic (PLEG): container finished" podID="cff713aa-db97-4366-ae04-c2bb57261598" containerID="8d93b706b96f780e2f683a349de87f3ca93b7916ef7c7d0732adbab88199a671" exitCode=0 Oct 30 17:23:41.551659 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.551572 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" Oct 30 17:23:41.551659 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.551608 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" event={"ID":"cff713aa-db97-4366-ae04-c2bb57261598","Type":"ContainerDied","Data":"8d93b706b96f780e2f683a349de87f3ca93b7916ef7c7d0732adbab88199a671"} Oct 30 17:23:41.551791 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.551657 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj" event={"ID":"cff713aa-db97-4366-ae04-c2bb57261598","Type":"ContainerDied","Data":"2a4cd9099533727e77eff2a9baa064647c3e8e710f40c215d0e1bde65116ebaa"} Oct 30 17:23:41.600358 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.600284 4861 scope.go:122] "RemoveContainer" containerID="b44ce697cbe329b4ec99d74e148212d8df50a00b4a8f7c93c528406d9e737dc6" Oct 30 17:23:41.600870 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:23:41.600838 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b44ce697cbe329b4ec99d74e148212d8df50a00b4a8f7c93c528406d9e737dc6\": container with ID starting with b44ce697cbe329b4ec99d74e148212d8df50a00b4a8f7c93c528406d9e737dc6 not found: ID does not exist" containerID="b44ce697cbe329b4ec99d74e148212d8df50a00b4a8f7c93c528406d9e737dc6" Oct 30 17:23:41.601025 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.600878 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b44ce697cbe329b4ec99d74e148212d8df50a00b4a8f7c93c528406d9e737dc6"} err="failed to get container status \"b44ce697cbe329b4ec99d74e148212d8df50a00b4a8f7c93c528406d9e737dc6\": rpc error: code = NotFound desc = could not find container \"b44ce697cbe329b4ec99d74e148212d8df50a00b4a8f7c93c528406d9e737dc6\": container with ID starting with b44ce697cbe329b4ec99d74e148212d8df50a00b4a8f7c93c528406d9e737dc6 not found: ID does not exist" Oct 30 17:23:41.601025 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.600900 4861 scope.go:122] "RemoveContainer" containerID="8d93b706b96f780e2f683a349de87f3ca93b7916ef7c7d0732adbab88199a671" Oct 30 17:23:41.613798 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.613746 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj"] Oct 30 17:23:41.616048 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.616019 4861 scope.go:122] "RemoveContainer" containerID="8d93b706b96f780e2f683a349de87f3ca93b7916ef7c7d0732adbab88199a671" Oct 30 17:23:41.616417 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:23:41.616391 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d93b706b96f780e2f683a349de87f3ca93b7916ef7c7d0732adbab88199a671\": container with ID starting with 8d93b706b96f780e2f683a349de87f3ca93b7916ef7c7d0732adbab88199a671 not found: ID does not exist" containerID="8d93b706b96f780e2f683a349de87f3ca93b7916ef7c7d0732adbab88199a671" Oct 30 17:23:41.616513 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.616429 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d93b706b96f780e2f683a349de87f3ca93b7916ef7c7d0732adbab88199a671"} err="failed to get container status \"8d93b706b96f780e2f683a349de87f3ca93b7916ef7c7d0732adbab88199a671\": rpc error: code = NotFound desc = could not find container \"8d93b706b96f780e2f683a349de87f3ca93b7916ef7c7d0732adbab88199a671\": container with ID starting with 8d93b706b96f780e2f683a349de87f3ca93b7916ef7c7d0732adbab88199a671 not found: ID does not exist" Oct 30 17:23:41.625465 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.625288 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-dd715e-7bd78fb4b-mr7xj"] Oct 30 17:23:41.652911 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.649594 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5"] Oct 30 17:23:41.656024 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.655956 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-dd715e-6999b5575f-grwm5"] Oct 30 17:23:41.770959 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.770923 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" Oct 30 17:23:41.809153 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.809124 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c8b7b8-b899-4303-870f-dce7e002fb42" path="/var/lib/kubelet/pods/26c8b7b8-b899-4303-870f-dce7e002fb42/volumes" Oct 30 17:23:41.809604 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.809590 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff713aa-db97-4366-ae04-c2bb57261598" path="/var/lib/kubelet/pods/cff713aa-db97-4366-ae04-c2bb57261598/volumes" Oct 30 17:23:41.912887 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.912757 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1995227-e125-48f5-8bf8-ff46ae4fc16e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1995227-e125-48f5-8bf8-ff46ae4fc16e-plugins-config-volume\") pod \"b1995227-e125-48f5-8bf8-ff46ae4fc16e\" (UID: \"b1995227-e125-48f5-8bf8-ff46ae4fc16e\") " Oct 30 17:23:41.913060 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.913009 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1995227-e125-48f5-8bf8-ff46ae4fc16e-kube-api-access-5jqbl\" (UniqueName: \"kubernetes.io/projected/b1995227-e125-48f5-8bf8-ff46ae4fc16e-kube-api-access-5jqbl\") pod \"b1995227-e125-48f5-8bf8-ff46ae4fc16e\" (UID: \"b1995227-e125-48f5-8bf8-ff46ae4fc16e\") " Oct 30 17:23:41.915051 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.915012 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1995227-e125-48f5-8bf8-ff46ae4fc16e-plugins-config-volume" pod "b1995227-e125-48f5-8bf8-ff46ae4fc16e" (UID: "b1995227-e125-48f5-8bf8-ff46ae4fc16e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:41.915195 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.915152 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1995227-e125-48f5-8bf8-ff46ae4fc16e-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:41.916207 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:41.916185 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1995227-e125-48f5-8bf8-ff46ae4fc16e-kube-api-access-5jqbl" pod "b1995227-e125-48f5-8bf8-ff46ae4fc16e" (UID: "b1995227-e125-48f5-8bf8-ff46ae4fc16e"). InnerVolumeSpecName "kube-api-access-5jqbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:42.020362 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:42.020287 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jqbl\" (UniqueName: \"kubernetes.io/projected/b1995227-e125-48f5-8bf8-ff46ae4fc16e-kube-api-access-5jqbl\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:42.562190 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:42.562155 4861 generic.go:356] "Generic (PLEG): container finished" podID="b1995227-e125-48f5-8bf8-ff46ae4fc16e" containerID="8f7d664e5d4c3424f82afa24419d27a9efea402cd3c4b898ea5080a6a322422d" exitCode=0 Oct 30 17:23:42.562190 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:42.562193 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" event={"ID":"b1995227-e125-48f5-8bf8-ff46ae4fc16e","Type":"ContainerDied","Data":"8f7d664e5d4c3424f82afa24419d27a9efea402cd3c4b898ea5080a6a322422d"} Oct 30 17:23:42.562739 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:42.562230 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" event={"ID":"b1995227-e125-48f5-8bf8-ff46ae4fc16e","Type":"ContainerDied","Data":"61769506548c843650a2101f7a9f93c66b117fc1d2a77b446f0432509c6f93a0"} Oct 30 17:23:42.562739 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:42.562253 4861 scope.go:122] "RemoveContainer" containerID="8f7d664e5d4c3424f82afa24419d27a9efea402cd3c4b898ea5080a6a322422d" Oct 30 17:23:42.562739 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:42.562196 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz" Oct 30 17:23:42.574111 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:42.574095 4861 scope.go:122] "RemoveContainer" containerID="8f7d664e5d4c3424f82afa24419d27a9efea402cd3c4b898ea5080a6a322422d" Oct 30 17:23:42.574339 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:23:42.574322 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7d664e5d4c3424f82afa24419d27a9efea402cd3c4b898ea5080a6a322422d\": container with ID starting with 8f7d664e5d4c3424f82afa24419d27a9efea402cd3c4b898ea5080a6a322422d not found: ID does not exist" containerID="8f7d664e5d4c3424f82afa24419d27a9efea402cd3c4b898ea5080a6a322422d" Oct 30 17:23:42.574405 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:42.574344 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7d664e5d4c3424f82afa24419d27a9efea402cd3c4b898ea5080a6a322422d"} err="failed to get container status \"8f7d664e5d4c3424f82afa24419d27a9efea402cd3c4b898ea5080a6a322422d\": rpc error: code = NotFound desc = could not find container \"8f7d664e5d4c3424f82afa24419d27a9efea402cd3c4b898ea5080a6a322422d\": container with ID starting with 8f7d664e5d4c3424f82afa24419d27a9efea402cd3c4b898ea5080a6a322422d not found: ID does not exist" Oct 30 17:23:42.627141 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:42.627113 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz"] Oct 30 17:23:42.640446 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:42.640423 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6e08bd-759bccbf66-m49sz"] Oct 30 17:23:43.811818 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:43.811781 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1995227-e125-48f5-8bf8-ff46ae4fc16e" path="/var/lib/kubelet/pods/b1995227-e125-48f5-8bf8-ff46ae4fc16e/volumes" Oct 30 17:23:45.833837 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:45.833800 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" podUID="6013ebf2-30f0-42a9-a31f-e66795df2b56" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:23:47.917249 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:47.917212 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b"] Oct 30 17:23:50.292253 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:23:50.292214 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:23:52.244438 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.244397 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" podUID="3141d3a0-8e3a-4232-b9bc-2de718b19b1e" containerName="main" containerID="cri-o://d6af706071d12358749e4c562e238201ec770a0a0eb5ea74931fc3614afab20e" gracePeriod=45 Oct 30 17:23:52.276079 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.276047 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" podUID="3141d3a0-8e3a-4232-b9bc-2de718b19b1e" containerName="main" probeResult="failure" output="Get \"https://10.132.1.42:8000/ready\": dial tcp 10.132.1.42:8000: connect: connection refused" Oct 30 17:23:52.544904 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.544880 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:23:52.588900 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.588875 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-kube-api-access-gjkhf\" (UniqueName: \"kubernetes.io/projected/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-kube-api-access-gjkhf\") pod \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " Oct 30 17:23:52.589073 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.589014 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-model-cache\") pod \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " Oct 30 17:23:52.589146 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.589093 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-tls-certs\" (UniqueName: \"kubernetes.io/secret/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-tls-certs\") pod \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " Oct 30 17:23:52.589270 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.589241 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-dshm\") pod \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " Oct 30 17:23:52.589388 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.589286 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-home\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-home\") pod \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " Oct 30 17:23:52.589388 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.589312 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-model-cache" pod "3141d3a0-8e3a-4232-b9bc-2de718b19b1e" (UID: "3141d3a0-8e3a-4232-b9bc-2de718b19b1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:52.589388 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.589340 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-tmp-dir\") pod \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\" (UID: \"3141d3a0-8e3a-4232-b9bc-2de718b19b1e\") " Oct 30 17:23:52.589545 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.589526 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-home" pod "3141d3a0-8e3a-4232-b9bc-2de718b19b1e" (UID: "3141d3a0-8e3a-4232-b9bc-2de718b19b1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:52.589652 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.589629 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-tmp-dir" pod "3141d3a0-8e3a-4232-b9bc-2de718b19b1e" (UID: "3141d3a0-8e3a-4232-b9bc-2de718b19b1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:52.590188 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.590160 4861 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:52.590292 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.590190 4861 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:52.590292 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.590204 4861 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:52.591916 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.591889 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-dshm" pod "3141d3a0-8e3a-4232-b9bc-2de718b19b1e" (UID: "3141d3a0-8e3a-4232-b9bc-2de718b19b1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:23:52.592025 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.591959 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-kube-api-access-gjkhf" pod "3141d3a0-8e3a-4232-b9bc-2de718b19b1e" (UID: "3141d3a0-8e3a-4232-b9bc-2de718b19b1e"). InnerVolumeSpecName "kube-api-access-gjkhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:52.592025 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.591964 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-tls-certs" pod "3141d3a0-8e3a-4232-b9bc-2de718b19b1e" (UID: "3141d3a0-8e3a-4232-b9bc-2de718b19b1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:52.691788 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.691757 4861 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:52.691788 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.691783 4861 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:52.691788 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.691793 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjkhf\" (UniqueName: \"kubernetes.io/projected/3141d3a0-8e3a-4232-b9bc-2de718b19b1e-kube-api-access-gjkhf\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:52.730782 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.730754 4861 scope.go:122] "RemoveContainer" containerID="2c26e348f0500d26b63327cc84a24fa6ac741c54f142196b671bc37317793858" Oct 30 17:23:52.753023 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.752989 4861 generic.go:356] "Generic (PLEG): container finished" podID="3141d3a0-8e3a-4232-b9bc-2de718b19b1e" containerID="d6af706071d12358749e4c562e238201ec770a0a0eb5ea74931fc3614afab20e" exitCode=0 Oct 30 17:23:52.753145 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.753042 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" Oct 30 17:23:52.753145 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.753067 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" event={"ID":"3141d3a0-8e3a-4232-b9bc-2de718b19b1e","Type":"ContainerDied","Data":"d6af706071d12358749e4c562e238201ec770a0a0eb5ea74931fc3614afab20e"} Oct 30 17:23:52.753145 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.753118 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8" event={"ID":"3141d3a0-8e3a-4232-b9bc-2de718b19b1e","Type":"ContainerDied","Data":"1c59d4d290bbe4f0761876187276db4542ba3b7ce20e0d0c4aef9fa876759371"} Oct 30 17:23:52.753145 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.753138 4861 scope.go:122] "RemoveContainer" containerID="d6af706071d12358749e4c562e238201ec770a0a0eb5ea74931fc3614afab20e" Oct 30 17:23:52.765861 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.765709 4861 scope.go:122] "RemoveContainer" containerID="d6af706071d12358749e4c562e238201ec770a0a0eb5ea74931fc3614afab20e" Oct 30 17:23:52.766020 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:23:52.765999 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6af706071d12358749e4c562e238201ec770a0a0eb5ea74931fc3614afab20e\": container with ID starting with d6af706071d12358749e4c562e238201ec770a0a0eb5ea74931fc3614afab20e not found: ID does not exist" containerID="d6af706071d12358749e4c562e238201ec770a0a0eb5ea74931fc3614afab20e" Oct 30 17:23:52.766072 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.766024 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6af706071d12358749e4c562e238201ec770a0a0eb5ea74931fc3614afab20e"} err="failed to get container status \"d6af706071d12358749e4c562e238201ec770a0a0eb5ea74931fc3614afab20e\": rpc error: code = NotFound desc = could not find container \"d6af706071d12358749e4c562e238201ec770a0a0eb5ea74931fc3614afab20e\": container with ID starting with d6af706071d12358749e4c562e238201ec770a0a0eb5ea74931fc3614afab20e not found: ID does not exist" Oct 30 17:23:52.811199 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.811161 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8"] Oct 30 17:23:52.814990 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.814970 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4c6596/test-model-4c6596-kserve-7f759ff96f-svdn8"] Oct 30 17:23:52.918520 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:52.918474 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" podUID="af41a080-c055-4886-8143-d586a24b1164" containerName="payload-processing" containerID="cri-o://ced68447fdb3b5770867fa96982bd4c05daea2681675b7e4e9db595072ad9b6b" gracePeriod=25 Oct 30 17:23:53.573114 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:53.573078 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" podUID="af41a080-c055-4886-8143-d586a24b1164" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:23:53.810001 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:53.809967 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3141d3a0-8e3a-4232-b9bc-2de718b19b1e" path="/var/lib/kubelet/pods/3141d3a0-8e3a-4232-b9bc-2de718b19b1e/volumes" Oct 30 17:23:54.464368 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:54.464343 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" Oct 30 17:23:54.515988 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:54.515959 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6013ebf2-30f0-42a9-a31f-e66795df2b56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6013ebf2-30f0-42a9-a31f-e66795df2b56-plugins-config-volume\") pod \"6013ebf2-30f0-42a9-a31f-e66795df2b56\" (UID: \"6013ebf2-30f0-42a9-a31f-e66795df2b56\") " Oct 30 17:23:54.516144 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:54.516006 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6013ebf2-30f0-42a9-a31f-e66795df2b56-kube-api-access-tzsqt\" (UniqueName: \"kubernetes.io/projected/6013ebf2-30f0-42a9-a31f-e66795df2b56-kube-api-access-tzsqt\") pod \"6013ebf2-30f0-42a9-a31f-e66795df2b56\" (UID: \"6013ebf2-30f0-42a9-a31f-e66795df2b56\") " Oct 30 17:23:54.516331 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:54.516309 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6013ebf2-30f0-42a9-a31f-e66795df2b56-plugins-config-volume" pod "6013ebf2-30f0-42a9-a31f-e66795df2b56" (UID: "6013ebf2-30f0-42a9-a31f-e66795df2b56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:54.516660 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:54.516641 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6013ebf2-30f0-42a9-a31f-e66795df2b56-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:54.524959 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:54.524932 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6013ebf2-30f0-42a9-a31f-e66795df2b56-kube-api-access-tzsqt" pod "6013ebf2-30f0-42a9-a31f-e66795df2b56" (UID: "6013ebf2-30f0-42a9-a31f-e66795df2b56"). InnerVolumeSpecName "kube-api-access-tzsqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:54.618467 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:54.618425 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzsqt\" (UniqueName: \"kubernetes.io/projected/6013ebf2-30f0-42a9-a31f-e66795df2b56-kube-api-access-tzsqt\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:54.771420 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:54.771388 4861 generic.go:356] "Generic (PLEG): container finished" podID="6013ebf2-30f0-42a9-a31f-e66795df2b56" containerID="0d434a8083452c5d5d095456075efecc78d7926662d0d95d205821c8ba01b6b5" exitCode=0 Oct 30 17:23:54.771595 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:54.771452 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" Oct 30 17:23:54.771595 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:54.771478 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" event={"ID":"6013ebf2-30f0-42a9-a31f-e66795df2b56","Type":"ContainerDied","Data":"0d434a8083452c5d5d095456075efecc78d7926662d0d95d205821c8ba01b6b5"} Oct 30 17:23:54.771595 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:54.771580 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc" event={"ID":"6013ebf2-30f0-42a9-a31f-e66795df2b56","Type":"ContainerDied","Data":"d6fd2beacaf67907a50c30c88cb353941f56dd4276a5839b35344240583411e0"} Oct 30 17:23:54.771707 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:54.771613 4861 scope.go:122] "RemoveContainer" containerID="0d434a8083452c5d5d095456075efecc78d7926662d0d95d205821c8ba01b6b5" Oct 30 17:23:54.784827 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:54.784807 4861 scope.go:122] "RemoveContainer" containerID="0d434a8083452c5d5d095456075efecc78d7926662d0d95d205821c8ba01b6b5" Oct 30 17:23:54.785124 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:23:54.785101 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d434a8083452c5d5d095456075efecc78d7926662d0d95d205821c8ba01b6b5\": container with ID starting with 0d434a8083452c5d5d095456075efecc78d7926662d0d95d205821c8ba01b6b5 not found: ID does not exist" containerID="0d434a8083452c5d5d095456075efecc78d7926662d0d95d205821c8ba01b6b5" Oct 30 17:23:54.785206 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:54.785132 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d434a8083452c5d5d095456075efecc78d7926662d0d95d205821c8ba01b6b5"} err="failed to get container status \"0d434a8083452c5d5d095456075efecc78d7926662d0d95d205821c8ba01b6b5\": rpc error: code = NotFound desc = could not find container \"0d434a8083452c5d5d095456075efecc78d7926662d0d95d205821c8ba01b6b5\": container with ID starting with 0d434a8083452c5d5d095456075efecc78d7926662d0d95d205821c8ba01b6b5 not found: ID does not exist" Oct 30 17:23:54.834256 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:54.834233 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc"] Oct 30 17:23:54.847732 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:54.847697 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e15d1f-6dc676d975-p7hpc"] Oct 30 17:23:55.808630 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:55.808585 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6013ebf2-30f0-42a9-a31f-e66795df2b56" path="/var/lib/kubelet/pods/6013ebf2-30f0-42a9-a31f-e66795df2b56/volumes" Oct 30 17:23:58.085879 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.085846 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799"] Oct 30 17:23:58.086329 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.086137 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" podUID="3c432eb4-2ac7-424b-9a55-6d21233b4807" containerName="maas-api" containerID="cri-o://9fb0fe9310c16121d51c80b91edd77cf760a844a5becf46d7f96c183016e5a76" gracePeriod=30 Oct 30 17:23:58.462246 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.462221 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:23:58.585749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.583872 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c432eb4-2ac7-424b-9a55-6d21233b4807-kube-api-access-9fjv5\" (UniqueName: \"kubernetes.io/projected/3c432eb4-2ac7-424b-9a55-6d21233b4807-kube-api-access-9fjv5\") pod \"3c432eb4-2ac7-424b-9a55-6d21233b4807\" (UID: \"3c432eb4-2ac7-424b-9a55-6d21233b4807\") " Oct 30 17:23:58.585749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.584196 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c432eb4-2ac7-424b-9a55-6d21233b4807-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c432eb4-2ac7-424b-9a55-6d21233b4807-metrics-certs\") pod \"3c432eb4-2ac7-424b-9a55-6d21233b4807\" (UID: \"3c432eb4-2ac7-424b-9a55-6d21233b4807\") " Oct 30 17:23:58.585749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.584402 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c432eb4-2ac7-424b-9a55-6d21233b4807-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c432eb4-2ac7-424b-9a55-6d21233b4807-openshift-service-ca\") pod \"3c432eb4-2ac7-424b-9a55-6d21233b4807\" (UID: \"3c432eb4-2ac7-424b-9a55-6d21233b4807\") " Oct 30 17:23:58.585749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.584435 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c432eb4-2ac7-424b-9a55-6d21233b4807-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c432eb4-2ac7-424b-9a55-6d21233b4807-maas-api-tls\") pod \"3c432eb4-2ac7-424b-9a55-6d21233b4807\" (UID: \"3c432eb4-2ac7-424b-9a55-6d21233b4807\") " Oct 30 17:23:58.608762 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.588433 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c432eb4-2ac7-424b-9a55-6d21233b4807-openshift-service-ca" pod "3c432eb4-2ac7-424b-9a55-6d21233b4807" (UID: "3c432eb4-2ac7-424b-9a55-6d21233b4807"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:58.608762 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.598925 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c432eb4-2ac7-424b-9a55-6d21233b4807-kube-api-access-9fjv5" pod "3c432eb4-2ac7-424b-9a55-6d21233b4807" (UID: "3c432eb4-2ac7-424b-9a55-6d21233b4807"). InnerVolumeSpecName "kube-api-access-9fjv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:58.638749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.634535 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c432eb4-2ac7-424b-9a55-6d21233b4807-maas-api-tls" pod "3c432eb4-2ac7-424b-9a55-6d21233b4807" (UID: "3c432eb4-2ac7-424b-9a55-6d21233b4807"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:58.638749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.638181 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c432eb4-2ac7-424b-9a55-6d21233b4807-metrics-certs" pod "3c432eb4-2ac7-424b-9a55-6d21233b4807" (UID: "3c432eb4-2ac7-424b-9a55-6d21233b4807"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:58.686341 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.686261 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c432eb4-2ac7-424b-9a55-6d21233b4807-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:58.686341 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.686298 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c432eb4-2ac7-424b-9a55-6d21233b4807-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:58.686341 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.686313 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fjv5\" (UniqueName: \"kubernetes.io/projected/3c432eb4-2ac7-424b-9a55-6d21233b4807-kube-api-access-9fjv5\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:58.686341 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.686325 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c432eb4-2ac7-424b-9a55-6d21233b4807-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:23:58.809700 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.809667 4861 generic.go:356] "Generic (PLEG): container finished" podID="3c432eb4-2ac7-424b-9a55-6d21233b4807" containerID="9fb0fe9310c16121d51c80b91edd77cf760a844a5becf46d7f96c183016e5a76" exitCode=0 Oct 30 17:23:58.809909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.809711 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" Oct 30 17:23:58.809909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.809753 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" event={"ID":"3c432eb4-2ac7-424b-9a55-6d21233b4807","Type":"ContainerDied","Data":"9fb0fe9310c16121d51c80b91edd77cf760a844a5becf46d7f96c183016e5a76"} Oct 30 17:23:58.809909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.809813 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799" event={"ID":"3c432eb4-2ac7-424b-9a55-6d21233b4807","Type":"ContainerDied","Data":"976d83bd59eeaf5879bd8726008a45c14b34633096b81f7affdd9e16d7125651"} Oct 30 17:23:58.809909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.809846 4861 scope.go:122] "RemoveContainer" containerID="9fb0fe9310c16121d51c80b91edd77cf760a844a5becf46d7f96c183016e5a76" Oct 30 17:23:58.828446 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.828425 4861 scope.go:122] "RemoveContainer" containerID="9fb0fe9310c16121d51c80b91edd77cf760a844a5becf46d7f96c183016e5a76" Oct 30 17:23:58.828845 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:23:58.828825 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb0fe9310c16121d51c80b91edd77cf760a844a5becf46d7f96c183016e5a76\": container with ID starting with 9fb0fe9310c16121d51c80b91edd77cf760a844a5becf46d7f96c183016e5a76 not found: ID does not exist" containerID="9fb0fe9310c16121d51c80b91edd77cf760a844a5becf46d7f96c183016e5a76" Oct 30 17:23:58.828943 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.828850 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb0fe9310c16121d51c80b91edd77cf760a844a5becf46d7f96c183016e5a76"} err="failed to get container status \"9fb0fe9310c16121d51c80b91edd77cf760a844a5becf46d7f96c183016e5a76\": rpc error: code = NotFound desc = could not find container \"9fb0fe9310c16121d51c80b91edd77cf760a844a5becf46d7f96c183016e5a76\": container with ID starting with 9fb0fe9310c16121d51c80b91edd77cf760a844a5becf46d7f96c183016e5a76 not found: ID does not exist" Oct 30 17:23:58.868584 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.868549 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799"] Oct 30 17:23:58.882281 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:58.882257 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e15d1f-67db6b66-z5799"] Oct 30 17:23:59.809291 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:23:59.809257 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c432eb4-2ac7-424b-9a55-6d21233b4807" path="/var/lib/kubelet/pods/3c432eb4-2ac7-424b-9a55-6d21233b4807/volumes" Oct 30 17:24:03.572910 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:03.572871 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" podUID="af41a080-c055-4886-8143-d586a24b1164" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:24:13.197406 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:13.197334 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" Oct 30 17:24:13.291152 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:13.291117 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af41a080-c055-4886-8143-d586a24b1164-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af41a080-c055-4886-8143-d586a24b1164-plugins-config-volume\") pod \"af41a080-c055-4886-8143-d586a24b1164\" (UID: \"af41a080-c055-4886-8143-d586a24b1164\") " Oct 30 17:24:13.291370 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:13.291206 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af41a080-c055-4886-8143-d586a24b1164-kube-api-access-mplhs\" (UniqueName: \"kubernetes.io/projected/af41a080-c055-4886-8143-d586a24b1164-kube-api-access-mplhs\") pod \"af41a080-c055-4886-8143-d586a24b1164\" (UID: \"af41a080-c055-4886-8143-d586a24b1164\") " Oct 30 17:24:13.291636 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:13.291605 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af41a080-c055-4886-8143-d586a24b1164-plugins-config-volume" pod "af41a080-c055-4886-8143-d586a24b1164" (UID: "af41a080-c055-4886-8143-d586a24b1164"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:13.292125 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:13.292097 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af41a080-c055-4886-8143-d586a24b1164-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:13.293669 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:13.293638 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af41a080-c055-4886-8143-d586a24b1164-kube-api-access-mplhs" pod "af41a080-c055-4886-8143-d586a24b1164" (UID: "af41a080-c055-4886-8143-d586a24b1164"). InnerVolumeSpecName "kube-api-access-mplhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:13.393227 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:13.393191 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mplhs\" (UniqueName: \"kubernetes.io/projected/af41a080-c055-4886-8143-d586a24b1164-kube-api-access-mplhs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:13.930943 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:13.930902 4861 generic.go:356] "Generic (PLEG): container finished" podID="af41a080-c055-4886-8143-d586a24b1164" containerID="ced68447fdb3b5770867fa96982bd4c05daea2681675b7e4e9db595072ad9b6b" exitCode=0 Oct 30 17:24:13.931156 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:13.930949 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" Oct 30 17:24:13.931156 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:13.930991 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" event={"ID":"af41a080-c055-4886-8143-d586a24b1164","Type":"ContainerDied","Data":"ced68447fdb3b5770867fa96982bd4c05daea2681675b7e4e9db595072ad9b6b"} Oct 30 17:24:13.931156 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:13.931056 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b" event={"ID":"af41a080-c055-4886-8143-d586a24b1164","Type":"ContainerDied","Data":"6e004c5c52ae191303c5f6e89cc2fe794338af673b58af2315d898c5082677bd"} Oct 30 17:24:13.931156 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:13.931081 4861 scope.go:122] "RemoveContainer" containerID="ced68447fdb3b5770867fa96982bd4c05daea2681675b7e4e9db595072ad9b6b" Oct 30 17:24:13.942277 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:13.942248 4861 scope.go:122] "RemoveContainer" containerID="ced68447fdb3b5770867fa96982bd4c05daea2681675b7e4e9db595072ad9b6b" Oct 30 17:24:13.942568 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:24:13.942549 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced68447fdb3b5770867fa96982bd4c05daea2681675b7e4e9db595072ad9b6b\": container with ID starting with ced68447fdb3b5770867fa96982bd4c05daea2681675b7e4e9db595072ad9b6b not found: ID does not exist" containerID="ced68447fdb3b5770867fa96982bd4c05daea2681675b7e4e9db595072ad9b6b" Oct 30 17:24:13.942623 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:13.942577 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced68447fdb3b5770867fa96982bd4c05daea2681675b7e4e9db595072ad9b6b"} err="failed to get container status \"ced68447fdb3b5770867fa96982bd4c05daea2681675b7e4e9db595072ad9b6b\": rpc error: code = NotFound desc = could not find container \"ced68447fdb3b5770867fa96982bd4c05daea2681675b7e4e9db595072ad9b6b\": container with ID starting with ced68447fdb3b5770867fa96982bd4c05daea2681675b7e4e9db595072ad9b6b not found: ID does not exist" Oct 30 17:24:13.989137 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:13.989110 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b"] Oct 30 17:24:14.002458 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:14.002427 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9afbb2-dbb6956dd-qh48b"] Oct 30 17:24:15.809198 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:15.809163 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af41a080-c055-4886-8143-d586a24b1164" path="/var/lib/kubelet/pods/af41a080-c055-4886-8143-d586a24b1164/volumes" Oct 30 17:24:20.291833 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:24:20.291787 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:24:25.757070 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:25.757025 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6"] Oct 30 17:24:25.757661 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:25.757468 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" podUID="99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6" containerName="maas-api" containerID="cri-o://d994cc3cd3ec4becd14ce3079ab82e125b864d44b038c36cd46b73dba877d084" gracePeriod=30 Oct 30 17:24:26.068639 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.068311 4861 generic.go:356] "Generic (PLEG): container finished" podID="99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6" containerID="d994cc3cd3ec4becd14ce3079ab82e125b864d44b038c36cd46b73dba877d084" exitCode=0 Oct 30 17:24:26.068639 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.068509 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" event={"ID":"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6","Type":"ContainerDied","Data":"d994cc3cd3ec4becd14ce3079ab82e125b864d44b038c36cd46b73dba877d084"} Oct 30 17:24:26.123006 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.122965 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 17:24:26.272124 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.272084 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h"] Oct 30 17:24:26.272406 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.272364 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" podUID="d252b794-0a34-4e07-94ee-2b1c7a2e40d0" containerName="maas-api" containerID="cri-o://7896dc664db90fde91f936fa9f139cac7f50043e4a805dc253d7fb7c80cdfdb9" gracePeriod=30 Oct 30 17:24:26.323612 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.323527 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-metrics-certs\") pod \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\" (UID: \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\") " Oct 30 17:24:26.323830 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.323711 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-maas-api-tls\") pod \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\" (UID: \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\") " Oct 30 17:24:26.323934 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.323836 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-kube-api-access-hs7h8\" (UniqueName: \"kubernetes.io/projected/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-kube-api-access-hs7h8\") pod \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\" (UID: \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\") " Oct 30 17:24:26.324347 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.324322 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-openshift-service-ca\") pod \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\" (UID: \"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6\") " Oct 30 17:24:26.325080 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.325048 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-openshift-service-ca" pod "99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6" (UID: "99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:26.325621 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.325465 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.327521 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.327493 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-kube-api-access-hs7h8" pod "99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6" (UID: "99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6"). InnerVolumeSpecName "kube-api-access-hs7h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:26.327710 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.327684 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-metrics-certs" pod "99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6" (UID: "99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:26.329821 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.329794 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-maas-api-tls" pod "99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6" (UID: "99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:26.445378 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.434174 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.445378 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.434218 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.445378 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.434234 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs7h8\" (UniqueName: \"kubernetes.io/projected/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6-kube-api-access-hs7h8\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.664074 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.663121 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:24:26.744458 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.744424 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-metrics-certs\") pod \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\" (UID: \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\") " Oct 30 17:24:26.744686 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.744663 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-kube-api-access-mbr5g\" (UniqueName: \"kubernetes.io/projected/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-kube-api-access-mbr5g\") pod \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\" (UID: \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\") " Oct 30 17:24:26.744801 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.744747 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-openshift-service-ca\") pod \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\" (UID: \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\") " Oct 30 17:24:26.744801 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.744786 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-maas-api-tls\") pod \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\" (UID: \"d252b794-0a34-4e07-94ee-2b1c7a2e40d0\") " Oct 30 17:24:26.745216 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.745168 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-openshift-service-ca" pod "d252b794-0a34-4e07-94ee-2b1c7a2e40d0" (UID: "d252b794-0a34-4e07-94ee-2b1c7a2e40d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:26.745689 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.745637 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.747278 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.747236 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-maas-api-tls" pod "d252b794-0a34-4e07-94ee-2b1c7a2e40d0" (UID: "d252b794-0a34-4e07-94ee-2b1c7a2e40d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:26.747395 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.747352 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-metrics-certs" pod "d252b794-0a34-4e07-94ee-2b1c7a2e40d0" (UID: "d252b794-0a34-4e07-94ee-2b1c7a2e40d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:26.747953 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.747926 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-kube-api-access-mbr5g" pod "d252b794-0a34-4e07-94ee-2b1c7a2e40d0" (UID: "d252b794-0a34-4e07-94ee-2b1c7a2e40d0"). InnerVolumeSpecName "kube-api-access-mbr5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:26.847534 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.847488 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.847534 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.847528 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:26.847990 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:26.847546 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbr5g\" (UniqueName: \"kubernetes.io/projected/d252b794-0a34-4e07-94ee-2b1c7a2e40d0-kube-api-access-mbr5g\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:27.077878 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:27.077843 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" Oct 30 17:24:27.078067 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:27.077877 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6" event={"ID":"99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6","Type":"ContainerDied","Data":"aaaa15acef7c857e313baf8c07cd177de66e7b6b83962ed565ccbe9853968ec8"} Oct 30 17:24:27.078067 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:27.077946 4861 scope.go:122] "RemoveContainer" containerID="d994cc3cd3ec4becd14ce3079ab82e125b864d44b038c36cd46b73dba877d084" Oct 30 17:24:27.079770 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:27.079694 4861 generic.go:356] "Generic (PLEG): container finished" podID="d252b794-0a34-4e07-94ee-2b1c7a2e40d0" containerID="7896dc664db90fde91f936fa9f139cac7f50043e4a805dc253d7fb7c80cdfdb9" exitCode=0 Oct 30 17:24:27.079770 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:27.079759 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" event={"ID":"d252b794-0a34-4e07-94ee-2b1c7a2e40d0","Type":"ContainerDied","Data":"7896dc664db90fde91f936fa9f139cac7f50043e4a805dc253d7fb7c80cdfdb9"} Oct 30 17:24:27.080004 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:27.079814 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" event={"ID":"d252b794-0a34-4e07-94ee-2b1c7a2e40d0","Type":"ContainerDied","Data":"b0061032a00f15396db094e7c101d007efb97250e8808becf3f599ec786a4c4b"} Oct 30 17:24:27.080004 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:27.079854 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h" Oct 30 17:24:27.094541 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:27.094518 4861 scope.go:122] "RemoveContainer" containerID="7896dc664db90fde91f936fa9f139cac7f50043e4a805dc253d7fb7c80cdfdb9" Oct 30 17:24:27.105968 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:27.105922 4861 scope.go:122] "RemoveContainer" containerID="7896dc664db90fde91f936fa9f139cac7f50043e4a805dc253d7fb7c80cdfdb9" Oct 30 17:24:27.106263 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:24:27.106224 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7896dc664db90fde91f936fa9f139cac7f50043e4a805dc253d7fb7c80cdfdb9\": container with ID starting with 7896dc664db90fde91f936fa9f139cac7f50043e4a805dc253d7fb7c80cdfdb9 not found: ID does not exist" containerID="7896dc664db90fde91f936fa9f139cac7f50043e4a805dc253d7fb7c80cdfdb9" Oct 30 17:24:27.106382 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:27.106271 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7896dc664db90fde91f936fa9f139cac7f50043e4a805dc253d7fb7c80cdfdb9"} err="failed to get container status \"7896dc664db90fde91f936fa9f139cac7f50043e4a805dc253d7fb7c80cdfdb9\": rpc error: code = NotFound desc = could not find container \"7896dc664db90fde91f936fa9f139cac7f50043e4a805dc253d7fb7c80cdfdb9\": container with ID starting with 7896dc664db90fde91f936fa9f139cac7f50043e4a805dc253d7fb7c80cdfdb9 not found: ID does not exist" Oct 30 17:24:27.162333 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:27.162300 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6"] Oct 30 17:24:27.170522 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:27.167926 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8bff6e-6dd4c975dc-sb5l6"] Oct 30 17:24:27.206574 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:27.206515 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h"] Oct 30 17:24:27.210594 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:27.210564 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e08bd-775fc48bf-g2z6h"] Oct 30 17:24:27.820013 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:27.819973 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6" path="/var/lib/kubelet/pods/99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6/volumes" Oct 30 17:24:27.820575 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:27.820553 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d252b794-0a34-4e07-94ee-2b1c7a2e40d0" path="/var/lib/kubelet/pods/d252b794-0a34-4e07-94ee-2b1c7a2e40d0/volumes" Oct 30 17:24:28.703565 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:28.703519 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64"] Oct 30 17:24:28.704093 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:28.703761 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" podUID="6e40b662-7953-4e26-921b-afb268c58d8e" containerName="maas-api" containerID="cri-o://4ad2a86de5d5cd10e7c04c133dd006f4e09c5e245127554c12c3924591857845" gracePeriod=30 Oct 30 17:24:29.046270 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.046244 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:24:29.088286 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.088254 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e40b662-7953-4e26-921b-afb268c58d8e-kube-api-access-ctgw6\" (UniqueName: \"kubernetes.io/projected/6e40b662-7953-4e26-921b-afb268c58d8e-kube-api-access-ctgw6\") pod \"6e40b662-7953-4e26-921b-afb268c58d8e\" (UID: \"6e40b662-7953-4e26-921b-afb268c58d8e\") " Oct 30 17:24:29.088469 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.088293 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e40b662-7953-4e26-921b-afb268c58d8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e40b662-7953-4e26-921b-afb268c58d8e-maas-api-tls\") pod \"6e40b662-7953-4e26-921b-afb268c58d8e\" (UID: \"6e40b662-7953-4e26-921b-afb268c58d8e\") " Oct 30 17:24:29.088469 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.088325 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e40b662-7953-4e26-921b-afb268c58d8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e40b662-7953-4e26-921b-afb268c58d8e-openshift-service-ca\") pod \"6e40b662-7953-4e26-921b-afb268c58d8e\" (UID: \"6e40b662-7953-4e26-921b-afb268c58d8e\") " Oct 30 17:24:29.088469 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.088393 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e40b662-7953-4e26-921b-afb268c58d8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e40b662-7953-4e26-921b-afb268c58d8e-metrics-certs\") pod \"6e40b662-7953-4e26-921b-afb268c58d8e\" (UID: \"6e40b662-7953-4e26-921b-afb268c58d8e\") " Oct 30 17:24:29.088739 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.088695 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e40b662-7953-4e26-921b-afb268c58d8e-openshift-service-ca" pod "6e40b662-7953-4e26-921b-afb268c58d8e" (UID: "6e40b662-7953-4e26-921b-afb268c58d8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:29.089218 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.089193 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e40b662-7953-4e26-921b-afb268c58d8e-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:29.090932 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.090906 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e40b662-7953-4e26-921b-afb268c58d8e-kube-api-access-ctgw6" pod "6e40b662-7953-4e26-921b-afb268c58d8e" (UID: "6e40b662-7953-4e26-921b-afb268c58d8e"). InnerVolumeSpecName "kube-api-access-ctgw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:29.091067 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.091040 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e40b662-7953-4e26-921b-afb268c58d8e-metrics-certs" pod "6e40b662-7953-4e26-921b-afb268c58d8e" (UID: "6e40b662-7953-4e26-921b-afb268c58d8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:29.091129 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.091085 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e40b662-7953-4e26-921b-afb268c58d8e-maas-api-tls" pod "6e40b662-7953-4e26-921b-afb268c58d8e" (UID: "6e40b662-7953-4e26-921b-afb268c58d8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:29.102874 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.102841 4861 generic.go:356] "Generic (PLEG): container finished" podID="6e40b662-7953-4e26-921b-afb268c58d8e" containerID="4ad2a86de5d5cd10e7c04c133dd006f4e09c5e245127554c12c3924591857845" exitCode=0 Oct 30 17:24:29.102993 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.102949 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" Oct 30 17:24:29.103080 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.102981 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" event={"ID":"6e40b662-7953-4e26-921b-afb268c58d8e","Type":"ContainerDied","Data":"4ad2a86de5d5cd10e7c04c133dd006f4e09c5e245127554c12c3924591857845"} Oct 30 17:24:29.103080 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.103057 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64" event={"ID":"6e40b662-7953-4e26-921b-afb268c58d8e","Type":"ContainerDied","Data":"27b1a175c5c38bccbe6e32ef2f245ca015f25bda2f423cef86896c714a949c38"} Oct 30 17:24:29.103201 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.103091 4861 scope.go:122] "RemoveContainer" containerID="4ad2a86de5d5cd10e7c04c133dd006f4e09c5e245127554c12c3924591857845" Oct 30 17:24:29.122035 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.121545 4861 scope.go:122] "RemoveContainer" containerID="4ad2a86de5d5cd10e7c04c133dd006f4e09c5e245127554c12c3924591857845" Oct 30 17:24:29.122182 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:24:29.122060 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad2a86de5d5cd10e7c04c133dd006f4e09c5e245127554c12c3924591857845\": container with ID starting with 4ad2a86de5d5cd10e7c04c133dd006f4e09c5e245127554c12c3924591857845 not found: ID does not exist" containerID="4ad2a86de5d5cd10e7c04c133dd006f4e09c5e245127554c12c3924591857845" Oct 30 17:24:29.122182 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.122095 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad2a86de5d5cd10e7c04c133dd006f4e09c5e245127554c12c3924591857845"} err="failed to get container status \"4ad2a86de5d5cd10e7c04c133dd006f4e09c5e245127554c12c3924591857845\": rpc error: code = NotFound desc = could not find container \"4ad2a86de5d5cd10e7c04c133dd006f4e09c5e245127554c12c3924591857845\": container with ID starting with 4ad2a86de5d5cd10e7c04c133dd006f4e09c5e245127554c12c3924591857845 not found: ID does not exist" Oct 30 17:24:29.166047 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.165996 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64"] Oct 30 17:24:29.170400 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.170377 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9afbb2-7b74984664-vjj64"] Oct 30 17:24:29.190489 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.190461 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctgw6\" (UniqueName: \"kubernetes.io/projected/6e40b662-7953-4e26-921b-afb268c58d8e-kube-api-access-ctgw6\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:29.190489 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.190487 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e40b662-7953-4e26-921b-afb268c58d8e-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:29.190632 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.190497 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e40b662-7953-4e26-921b-afb268c58d8e-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:29.853183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:29.853146 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e40b662-7953-4e26-921b-afb268c58d8e" path="/var/lib/kubelet/pods/6e40b662-7953-4e26-921b-afb268c58d8e/volumes" Oct 30 17:24:36.983675 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:36.982166 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr"] Oct 30 17:24:36.983675 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:36.982433 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" podUID="ce90a319-7468-4c37-8065-01f08642df9d" containerName="istio-proxy" containerID="cri-o://9d5615a42ba0f48d5b3ddbe7a8f9f3e530fd819bb75e12c004371ec74da69ad5" gracePeriod=30 Oct 30 17:24:39.289531 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:39.289493 4861 patch_prober.go:29] interesting pod/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:24:39.289960 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:39.289556 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" podUID="ce90a319-7468-4c37-8065-01f08642df9d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:24:42.222058 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.222018 4861 generic.go:356] "Generic (PLEG): container finished" podID="ce90a319-7468-4c37-8065-01f08642df9d" containerID="9d5615a42ba0f48d5b3ddbe7a8f9f3e530fd819bb75e12c004371ec74da69ad5" exitCode=0 Oct 30 17:24:42.222556 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.222084 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" event={"ID":"ce90a319-7468-4c37-8065-01f08642df9d","Type":"ContainerDied","Data":"9d5615a42ba0f48d5b3ddbe7a8f9f3e530fd819bb75e12c004371ec74da69ad5"} Oct 30 17:24:42.222556 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.222148 4861 scope.go:122] "RemoveContainer" containerID="de35be27d45cdd31e9edbed0c278c82b2c295ef510c523af0ef06e4ddeb0f1db" Oct 30 17:24:42.279350 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.279325 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:24:42.407376 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.407345 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-workload-socket\") pod \"ce90a319-7468-4c37-8065-01f08642df9d\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " Oct 30 17:24:42.407532 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.407430 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-istio-envoy\") pod \"ce90a319-7468-4c37-8065-01f08642df9d\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " Oct 30 17:24:42.407532 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.407514 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce90a319-7468-4c37-8065-01f08642df9d-istio-token\" (UniqueName: \"kubernetes.io/projected/ce90a319-7468-4c37-8065-01f08642df9d-istio-token\") pod \"ce90a319-7468-4c37-8065-01f08642df9d\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " Oct 30 17:24:42.407638 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.407574 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce90a319-7468-4c37-8065-01f08642df9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce90a319-7468-4c37-8065-01f08642df9d-istiod-ca-cert\") pod \"ce90a319-7468-4c37-8065-01f08642df9d\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " Oct 30 17:24:42.407638 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.407615 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-workload-socket" pod "ce90a319-7468-4c37-8065-01f08642df9d" (UID: "ce90a319-7468-4c37-8065-01f08642df9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.407771 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.407707 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-workload-certs\") pod \"ce90a319-7468-4c37-8065-01f08642df9d\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " Oct 30 17:24:42.407881 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.407862 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce90a319-7468-4c37-8065-01f08642df9d-kube-api-access-vrjqg\" (UniqueName: \"kubernetes.io/projected/ce90a319-7468-4c37-8065-01f08642df9d-kube-api-access-vrjqg\") pod \"ce90a319-7468-4c37-8065-01f08642df9d\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " Oct 30 17:24:42.407944 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.407915 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce90a319-7468-4c37-8065-01f08642df9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce90a319-7468-4c37-8065-01f08642df9d-istio-podinfo\") pod \"ce90a319-7468-4c37-8065-01f08642df9d\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " Oct 30 17:24:42.407944 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.407930 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-workload-certs" pod "ce90a319-7468-4c37-8065-01f08642df9d" (UID: "ce90a319-7468-4c37-8065-01f08642df9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.408059 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.407968 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-istio-data\") pod \"ce90a319-7468-4c37-8065-01f08642df9d\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " Oct 30 17:24:42.408059 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.407976 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce90a319-7468-4c37-8065-01f08642df9d-istiod-ca-cert" pod "ce90a319-7468-4c37-8065-01f08642df9d" (UID: "ce90a319-7468-4c37-8065-01f08642df9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:42.408173 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.408076 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-credential-socket\") pod \"ce90a319-7468-4c37-8065-01f08642df9d\" (UID: \"ce90a319-7468-4c37-8065-01f08642df9d\") " Oct 30 17:24:42.408275 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.408234 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-istio-data" pod "ce90a319-7468-4c37-8065-01f08642df9d" (UID: "ce90a319-7468-4c37-8065-01f08642df9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.408497 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.408472 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-credential-socket" pod "ce90a319-7468-4c37-8065-01f08642df9d" (UID: "ce90a319-7468-4c37-8065-01f08642df9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.409124 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.409100 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce90a319-7468-4c37-8065-01f08642df9d-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.409193 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.409133 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.409193 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.409148 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.409193 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.409162 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.409193 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.409177 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.410217 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.410185 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce90a319-7468-4c37-8065-01f08642df9d-istio-token" pod "ce90a319-7468-4c37-8065-01f08642df9d" (UID: "ce90a319-7468-4c37-8065-01f08642df9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:42.410314 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.410255 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce90a319-7468-4c37-8065-01f08642df9d-kube-api-access-vrjqg" pod "ce90a319-7468-4c37-8065-01f08642df9d" (UID: "ce90a319-7468-4c37-8065-01f08642df9d"). InnerVolumeSpecName "kube-api-access-vrjqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:42.410314 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.410302 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-istio-envoy" pod "ce90a319-7468-4c37-8065-01f08642df9d" (UID: "ce90a319-7468-4c37-8065-01f08642df9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:24:42.410402 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.410331 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce90a319-7468-4c37-8065-01f08642df9d-istio-podinfo" pod "ce90a319-7468-4c37-8065-01f08642df9d" (UID: "ce90a319-7468-4c37-8065-01f08642df9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:24:42.510832 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.510713 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce90a319-7468-4c37-8065-01f08642df9d-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.510832 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.510780 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce90a319-7468-4c37-8065-01f08642df9d-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.510832 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.510790 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrjqg\" (UniqueName: \"kubernetes.io/projected/ce90a319-7468-4c37-8065-01f08642df9d-kube-api-access-vrjqg\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:42.510832 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:42.510799 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce90a319-7468-4c37-8065-01f08642df9d-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:43.230759 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:43.230697 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" Oct 30 17:24:43.231280 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:43.230766 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr" event={"ID":"ce90a319-7468-4c37-8065-01f08642df9d","Type":"ContainerDied","Data":"ef5f642c5308e373bdf6ecbdeb45dc4aaceb5b4678b6fecd6699aa924ba7bb6f"} Oct 30 17:24:43.231280 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:43.230829 4861 scope.go:122] "RemoveContainer" containerID="9d5615a42ba0f48d5b3ddbe7a8f9f3e530fd819bb75e12c004371ec74da69ad5" Oct 30 17:24:43.304161 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:43.304127 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr"] Oct 30 17:24:43.317350 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:43.317323 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9afbb2-openshift-default-99dc69c5-r8tmr"] Oct 30 17:24:43.809857 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:43.809824 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce90a319-7468-4c37-8065-01f08642df9d" path="/var/lib/kubelet/pods/ce90a319-7468-4c37-8065-01f08642df9d/volumes" Oct 30 17:24:45.241050 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:45.241011 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt"] Oct 30 17:24:45.275879 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:45.275847 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb"] Oct 30 17:24:46.057113 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:46.057073 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm"] Oct 30 17:24:48.073985 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:48.073949 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt"] Oct 30 17:24:48.107853 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:48.107820 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c"] Oct 30 17:24:50.241383 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:50.241338 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" podUID="6f5e2f42-7122-4574-b519-891d32498176" containerName="payload-processing" containerID="cri-o://adb282a7a2e1359c761b0a3012034b96b2958687ca4c0973967b20ae195b3177" gracePeriod=25 Oct 30 17:24:50.276314 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:50.276271 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" podUID="04fa61ca-48d1-4fb0-99c9-2781775881d4" containerName="payload-pre-processing" containerID="cri-o://b924af75adf400410178ee6f8627da9d0c286db86955f5e6a92c926b0aaa615f" gracePeriod=25 Oct 30 17:24:50.290909 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:24:50.290876 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:24:51.058470 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:51.058429 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" podUID="9345c3c1-50db-406b-81b6-202cb9b75496" containerName="payload-pre-processing" containerID="cri-o://f8ccfedb4595751530406be630724798a6bb2aca4dcf15d9769b5e00aca079ed" gracePeriod=25 Oct 30 17:24:51.776867 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:51.776835 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz"] Oct 30 17:24:51.777510 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:51.777455 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" podUID="a2c79376-2a02-4240-bde9-c58d20acb2f5" containerName="maas-api" containerID="cri-o://d06065b8f894a43160097a8bbcf058351b5c2a91d4e54872e89a2ef244523133" gracePeriod=30 Oct 30 17:24:52.109222 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.109193 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:24:52.208630 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.208287 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" podUID="9345c3c1-50db-406b-81b6-202cb9b75496" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:24:52.279179 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.279141 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2c79376-2a02-4240-bde9-c58d20acb2f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c79376-2a02-4240-bde9-c58d20acb2f5-metrics-certs\") pod \"a2c79376-2a02-4240-bde9-c58d20acb2f5\" (UID: \"a2c79376-2a02-4240-bde9-c58d20acb2f5\") " Oct 30 17:24:52.279400 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.279287 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c79376-2a02-4240-bde9-c58d20acb2f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c79376-2a02-4240-bde9-c58d20acb2f5-openshift-service-ca\") pod \"a2c79376-2a02-4240-bde9-c58d20acb2f5\" (UID: \"a2c79376-2a02-4240-bde9-c58d20acb2f5\") " Oct 30 17:24:52.279491 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.279479 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c79376-2a02-4240-bde9-c58d20acb2f5-kube-api-access-wxnsj\" (UniqueName: \"kubernetes.io/projected/a2c79376-2a02-4240-bde9-c58d20acb2f5-kube-api-access-wxnsj\") pod \"a2c79376-2a02-4240-bde9-c58d20acb2f5\" (UID: \"a2c79376-2a02-4240-bde9-c58d20acb2f5\") " Oct 30 17:24:52.279557 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.279509 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2c79376-2a02-4240-bde9-c58d20acb2f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c79376-2a02-4240-bde9-c58d20acb2f5-maas-api-tls\") pod \"a2c79376-2a02-4240-bde9-c58d20acb2f5\" (UID: \"a2c79376-2a02-4240-bde9-c58d20acb2f5\") " Oct 30 17:24:52.279784 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.279753 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c79376-2a02-4240-bde9-c58d20acb2f5-openshift-service-ca" pod "a2c79376-2a02-4240-bde9-c58d20acb2f5" (UID: "a2c79376-2a02-4240-bde9-c58d20acb2f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:52.280037 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.280017 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c79376-2a02-4240-bde9-c58d20acb2f5-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:52.282359 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.282287 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c79376-2a02-4240-bde9-c58d20acb2f5-metrics-certs" pod "a2c79376-2a02-4240-bde9-c58d20acb2f5" (UID: "a2c79376-2a02-4240-bde9-c58d20acb2f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:52.282663 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.282629 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c79376-2a02-4240-bde9-c58d20acb2f5-maas-api-tls" pod "a2c79376-2a02-4240-bde9-c58d20acb2f5" (UID: "a2c79376-2a02-4240-bde9-c58d20acb2f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:52.282663 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.282632 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c79376-2a02-4240-bde9-c58d20acb2f5-kube-api-access-wxnsj" pod "a2c79376-2a02-4240-bde9-c58d20acb2f5" (UID: "a2c79376-2a02-4240-bde9-c58d20acb2f5"). InnerVolumeSpecName "kube-api-access-wxnsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:52.321881 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.321799 4861 generic.go:356] "Generic (PLEG): container finished" podID="a2c79376-2a02-4240-bde9-c58d20acb2f5" containerID="d06065b8f894a43160097a8bbcf058351b5c2a91d4e54872e89a2ef244523133" exitCode=0 Oct 30 17:24:52.322107 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.322086 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" event={"ID":"a2c79376-2a02-4240-bde9-c58d20acb2f5","Type":"ContainerDied","Data":"d06065b8f894a43160097a8bbcf058351b5c2a91d4e54872e89a2ef244523133"} Oct 30 17:24:52.322181 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.322144 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" event={"ID":"a2c79376-2a02-4240-bde9-c58d20acb2f5","Type":"ContainerDied","Data":"2e6b59030187b035cc2e5dd739ee98a896cfa5ebd094d0302812d14ca8510945"} Oct 30 17:24:52.322240 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.322179 4861 scope.go:122] "RemoveContainer" containerID="d06065b8f894a43160097a8bbcf058351b5c2a91d4e54872e89a2ef244523133" Oct 30 17:24:52.322364 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.322349 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz" Oct 30 17:24:52.341372 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.341343 4861 scope.go:122] "RemoveContainer" containerID="d06065b8f894a43160097a8bbcf058351b5c2a91d4e54872e89a2ef244523133" Oct 30 17:24:52.342635 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:24:52.342603 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d06065b8f894a43160097a8bbcf058351b5c2a91d4e54872e89a2ef244523133\": container with ID starting with d06065b8f894a43160097a8bbcf058351b5c2a91d4e54872e89a2ef244523133 not found: ID does not exist" containerID="d06065b8f894a43160097a8bbcf058351b5c2a91d4e54872e89a2ef244523133" Oct 30 17:24:52.342747 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.342644 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d06065b8f894a43160097a8bbcf058351b5c2a91d4e54872e89a2ef244523133"} err="failed to get container status \"d06065b8f894a43160097a8bbcf058351b5c2a91d4e54872e89a2ef244523133\": rpc error: code = NotFound desc = could not find container \"d06065b8f894a43160097a8bbcf058351b5c2a91d4e54872e89a2ef244523133\": container with ID starting with d06065b8f894a43160097a8bbcf058351b5c2a91d4e54872e89a2ef244523133 not found: ID does not exist" Oct 30 17:24:52.381654 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.381630 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxnsj\" (UniqueName: \"kubernetes.io/projected/a2c79376-2a02-4240-bde9-c58d20acb2f5-kube-api-access-wxnsj\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:52.381654 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.381655 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2c79376-2a02-4240-bde9-c58d20acb2f5-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:52.381836 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.381667 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2c79376-2a02-4240-bde9-c58d20acb2f5-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:52.384341 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.384317 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz"] Oct 30 17:24:52.387982 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:52.387955 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-85eb87-97d96c6d-p97qz"] Oct 30 17:24:53.075003 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:53.074904 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" podUID="334f60fa-9e62-417d-8965-9a5a0f5bff47" containerName="payload-processing" containerID="cri-o://216bfe1202745b08df5791a3b127631d5815543355955ef426ac21b06972e5d1" gracePeriod=25 Oct 30 17:24:53.108669 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:53.108632 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" podUID="b56c9a33-1b9e-48dc-9ebe-d80778b20e35" containerName="payload-pre-processing" containerID="cri-o://433c4aee9dd83f4d3f9fb1c11446498cf3298240b667ac24a851257979ede425" gracePeriod=25 Oct 30 17:24:53.381553 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:53.371993 4861 generic.go:356] "Generic (PLEG): container finished" podID="334f60fa-9e62-417d-8965-9a5a0f5bff47" containerID="216bfe1202745b08df5791a3b127631d5815543355955ef426ac21b06972e5d1" exitCode=0 Oct 30 17:24:53.381553 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:53.372072 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" event={"ID":"334f60fa-9e62-417d-8965-9a5a0f5bff47","Type":"ContainerDied","Data":"216bfe1202745b08df5791a3b127631d5815543355955ef426ac21b06972e5d1"} Oct 30 17:24:53.419775 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:53.419743 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" Oct 30 17:24:53.497141 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:53.497108 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/334f60fa-9e62-417d-8965-9a5a0f5bff47-kube-api-access-6g8kk\" (UniqueName: \"kubernetes.io/projected/334f60fa-9e62-417d-8965-9a5a0f5bff47-kube-api-access-6g8kk\") pod \"334f60fa-9e62-417d-8965-9a5a0f5bff47\" (UID: \"334f60fa-9e62-417d-8965-9a5a0f5bff47\") " Oct 30 17:24:53.497360 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:53.497341 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/334f60fa-9e62-417d-8965-9a5a0f5bff47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334f60fa-9e62-417d-8965-9a5a0f5bff47-plugins-config-volume\") pod \"334f60fa-9e62-417d-8965-9a5a0f5bff47\" (UID: \"334f60fa-9e62-417d-8965-9a5a0f5bff47\") " Oct 30 17:24:53.497818 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:53.497786 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334f60fa-9e62-417d-8965-9a5a0f5bff47-plugins-config-volume" pod "334f60fa-9e62-417d-8965-9a5a0f5bff47" (UID: "334f60fa-9e62-417d-8965-9a5a0f5bff47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:53.498098 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:53.498032 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/334f60fa-9e62-417d-8965-9a5a0f5bff47-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:53.499645 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:53.499609 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334f60fa-9e62-417d-8965-9a5a0f5bff47-kube-api-access-6g8kk" pod "334f60fa-9e62-417d-8965-9a5a0f5bff47" (UID: "334f60fa-9e62-417d-8965-9a5a0f5bff47"). InnerVolumeSpecName "kube-api-access-6g8kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:53.599101 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:53.599065 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g8kk\" (UniqueName: \"kubernetes.io/projected/334f60fa-9e62-417d-8965-9a5a0f5bff47-kube-api-access-6g8kk\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:24:53.813342 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:53.813299 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c79376-2a02-4240-bde9-c58d20acb2f5" path="/var/lib/kubelet/pods/a2c79376-2a02-4240-bde9-c58d20acb2f5/volumes" Oct 30 17:24:54.380358 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:54.380321 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" event={"ID":"334f60fa-9e62-417d-8965-9a5a0f5bff47","Type":"ContainerDied","Data":"ae709990ae00512e3b40b629b20816285b56fa2954890c6e000344625d1d5bda"} Oct 30 17:24:54.380358 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:54.380353 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt" Oct 30 17:24:54.380864 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:54.380381 4861 scope.go:122] "RemoveContainer" containerID="216bfe1202745b08df5791a3b127631d5815543355955ef426ac21b06972e5d1" Oct 30 17:24:54.440360 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:54.440329 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt"] Oct 30 17:24:54.452604 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:54.452580 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4c6596-7b59bfb48-5tpdt"] Oct 30 17:24:55.808962 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:55.808924 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="334f60fa-9e62-417d-8965-9a5a0f5bff47" path="/var/lib/kubelet/pods/334f60fa-9e62-417d-8965-9a5a0f5bff47/volumes" Oct 30 17:24:55.862845 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:55.862810 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" podUID="04fa61ca-48d1-4fb0-99c9-2781775881d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:24:55.863025 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:24:55.862869 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" podUID="6f5e2f42-7122-4574-b519-891d32498176" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:00.203244 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:00.203208 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj"] Oct 30 17:25:00.203877 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:00.203466 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" podUID="f7dd7b91-6cba-4446-9f47-c536520520bf" containerName="istio-proxy" containerID="cri-o://0c140559e9dd1e7a300298d9a534075458e0f694d018c00be59ad78d53fae585" gracePeriod=30 Oct 30 17:25:02.169433 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:02.169396 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" podUID="b56c9a33-1b9e-48dc-9ebe-d80778b20e35" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:02.169870 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:02.169397 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" podUID="9345c3c1-50db-406b-81b6-202cb9b75496" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:02.479300 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:02.479266 4861 patch_prober.go:29] interesting pod/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:25:02.479516 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:02.479320 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" podUID="f7dd7b91-6cba-4446-9f47-c536520520bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:25:05.471944 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.471910 4861 generic.go:356] "Generic (PLEG): container finished" podID="f7dd7b91-6cba-4446-9f47-c536520520bf" containerID="0c140559e9dd1e7a300298d9a534075458e0f694d018c00be59ad78d53fae585" exitCode=0 Oct 30 17:25:05.472310 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.471985 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" event={"ID":"f7dd7b91-6cba-4446-9f47-c536520520bf","Type":"ContainerDied","Data":"0c140559e9dd1e7a300298d9a534075458e0f694d018c00be59ad78d53fae585"} Oct 30 17:25:05.472310 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.472107 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" event={"ID":"f7dd7b91-6cba-4446-9f47-c536520520bf","Type":"ContainerDied","Data":"f750e41005dc475168523f5edb0aa9f52253e03ceea1d69d8d7e889879a35cb0"} Oct 30 17:25:05.472310 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.472132 4861 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f750e41005dc475168523f5edb0aa9f52253e03ceea1d69d8d7e889879a35cb0" Oct 30 17:25:05.472310 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.472151 4861 scope.go:122] "RemoveContainer" containerID="ab7536ff7fc96f49373a624bcad025886f0c58be4ecab65fddef24ffdef42a04" Oct 30 17:25:05.497181 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.497158 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:25:05.594975 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.594940 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-credential-socket\") pod \"f7dd7b91-6cba-4446-9f47-c536520520bf\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " Oct 30 17:25:05.595169 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.595026 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-token\" (UniqueName: \"kubernetes.io/projected/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-token\") pod \"f7dd7b91-6cba-4446-9f47-c536520520bf\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " Oct 30 17:25:05.595169 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.595059 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7dd7b91-6cba-4446-9f47-c536520520bf-kube-api-access-5mzmm\" (UniqueName: \"kubernetes.io/projected/f7dd7b91-6cba-4446-9f47-c536520520bf-kube-api-access-5mzmm\") pod \"f7dd7b91-6cba-4446-9f47-c536520520bf\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " Oct 30 17:25:05.595169 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.595099 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-workload-socket\") pod \"f7dd7b91-6cba-4446-9f47-c536520520bf\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " Oct 30 17:25:05.595169 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.595128 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-envoy\") pod \"f7dd7b91-6cba-4446-9f47-c536520520bf\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " Oct 30 17:25:05.595169 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.595148 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-podinfo\") pod \"f7dd7b91-6cba-4446-9f47-c536520520bf\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " Oct 30 17:25:05.595169 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.595165 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-workload-certs\") pod \"f7dd7b91-6cba-4446-9f47-c536520520bf\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " Oct 30 17:25:05.595535 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.595230 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-credential-socket" pod "f7dd7b91-6cba-4446-9f47-c536520520bf" (UID: "f7dd7b91-6cba-4446-9f47-c536520520bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:05.595535 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.595250 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7dd7b91-6cba-4446-9f47-c536520520bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7dd7b91-6cba-4446-9f47-c536520520bf-istiod-ca-cert\") pod \"f7dd7b91-6cba-4446-9f47-c536520520bf\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " Oct 30 17:25:05.595535 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.595284 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-data\") pod \"f7dd7b91-6cba-4446-9f47-c536520520bf\" (UID: \"f7dd7b91-6cba-4446-9f47-c536520520bf\") " Oct 30 17:25:05.596047 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.595711 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-workload-socket" pod "f7dd7b91-6cba-4446-9f47-c536520520bf" (UID: "f7dd7b91-6cba-4446-9f47-c536520520bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:05.596047 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.596027 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-data" pod "f7dd7b91-6cba-4446-9f47-c536520520bf" (UID: "f7dd7b91-6cba-4446-9f47-c536520520bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:05.596047 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.596019 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:05.596321 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.596059 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:05.596386 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.596244 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-workload-certs" pod "f7dd7b91-6cba-4446-9f47-c536520520bf" (UID: "f7dd7b91-6cba-4446-9f47-c536520520bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:05.596569 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.596545 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7dd7b91-6cba-4446-9f47-c536520520bf-istiod-ca-cert" pod "f7dd7b91-6cba-4446-9f47-c536520520bf" (UID: "f7dd7b91-6cba-4446-9f47-c536520520bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:05.598414 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.598387 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-envoy" pod "f7dd7b91-6cba-4446-9f47-c536520520bf" (UID: "f7dd7b91-6cba-4446-9f47-c536520520bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:05.598539 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.598421 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-podinfo" pod "f7dd7b91-6cba-4446-9f47-c536520520bf" (UID: "f7dd7b91-6cba-4446-9f47-c536520520bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:25:05.598539 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.598452 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-token" pod "f7dd7b91-6cba-4446-9f47-c536520520bf" (UID: "f7dd7b91-6cba-4446-9f47-c536520520bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:05.598851 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.598823 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7dd7b91-6cba-4446-9f47-c536520520bf-kube-api-access-5mzmm" pod "f7dd7b91-6cba-4446-9f47-c536520520bf" (UID: "f7dd7b91-6cba-4446-9f47-c536520520bf"). InnerVolumeSpecName "kube-api-access-5mzmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:05.697457 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.697426 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mzmm\" (UniqueName: \"kubernetes.io/projected/f7dd7b91-6cba-4446-9f47-c536520520bf-kube-api-access-5mzmm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:05.697457 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.697454 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:05.697627 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.697464 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:05.697627 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.697473 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:05.697627 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.697481 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7dd7b91-6cba-4446-9f47-c536520520bf-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:05.697627 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.697489 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:05.697627 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.697497 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7dd7b91-6cba-4446-9f47-c536520520bf-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:05.870501 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.870460 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" podUID="6f5e2f42-7122-4574-b519-891d32498176" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:05.871180 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:05.871147 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" podUID="04fa61ca-48d1-4fb0-99c9-2781775881d4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:06.481821 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:06.481791 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj" Oct 30 17:25:06.585739 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:06.585701 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj"] Oct 30 17:25:06.589451 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:06.589429 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-85eb87-openshift-default-5b54cdb967-hpjrj"] Oct 30 17:25:07.808646 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:07.808600 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7dd7b91-6cba-4446-9f47-c536520520bf" path="/var/lib/kubelet/pods/f7dd7b91-6cba-4446-9f47-c536520520bf/volumes" Oct 30 17:25:10.516374 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.516279 4861 generic.go:356] "Generic (PLEG): container finished" podID="04fa61ca-48d1-4fb0-99c9-2781775881d4" containerID="b924af75adf400410178ee6f8627da9d0c286db86955f5e6a92c926b0aaa615f" exitCode=0 Oct 30 17:25:10.516897 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.516541 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" event={"ID":"04fa61ca-48d1-4fb0-99c9-2781775881d4","Type":"ContainerDied","Data":"b924af75adf400410178ee6f8627da9d0c286db86955f5e6a92c926b0aaa615f"} Oct 30 17:25:10.519388 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.519344 4861 generic.go:356] "Generic (PLEG): container finished" podID="6f5e2f42-7122-4574-b519-891d32498176" containerID="adb282a7a2e1359c761b0a3012034b96b2958687ca4c0973967b20ae195b3177" exitCode=0 Oct 30 17:25:10.519547 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.519476 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" event={"ID":"6f5e2f42-7122-4574-b519-891d32498176","Type":"ContainerDied","Data":"adb282a7a2e1359c761b0a3012034b96b2958687ca4c0973967b20ae195b3177"} Oct 30 17:25:10.567294 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.567270 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" Oct 30 17:25:10.589334 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.589311 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" Oct 30 17:25:10.669201 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.669172 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f5e2f42-7122-4574-b519-891d32498176-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f5e2f42-7122-4574-b519-891d32498176-plugins-config-volume\") pod \"6f5e2f42-7122-4574-b519-891d32498176\" (UID: \"6f5e2f42-7122-4574-b519-891d32498176\") " Oct 30 17:25:10.669406 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.669274 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5e2f42-7122-4574-b519-891d32498176-kube-api-access-9m4zk\" (UniqueName: \"kubernetes.io/projected/6f5e2f42-7122-4574-b519-891d32498176-kube-api-access-9m4zk\") pod \"6f5e2f42-7122-4574-b519-891d32498176\" (UID: \"6f5e2f42-7122-4574-b519-891d32498176\") " Oct 30 17:25:10.669486 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.669403 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fa61ca-48d1-4fb0-99c9-2781775881d4-kube-api-access-vgfl4\" (UniqueName: \"kubernetes.io/projected/04fa61ca-48d1-4fb0-99c9-2781775881d4-kube-api-access-vgfl4\") pod \"04fa61ca-48d1-4fb0-99c9-2781775881d4\" (UID: \"04fa61ca-48d1-4fb0-99c9-2781775881d4\") " Oct 30 17:25:10.669486 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.669475 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04fa61ca-48d1-4fb0-99c9-2781775881d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fa61ca-48d1-4fb0-99c9-2781775881d4-plugins-config-volume\") pod \"04fa61ca-48d1-4fb0-99c9-2781775881d4\" (UID: \"04fa61ca-48d1-4fb0-99c9-2781775881d4\") " Oct 30 17:25:10.669631 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.669604 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5e2f42-7122-4574-b519-891d32498176-plugins-config-volume" pod "6f5e2f42-7122-4574-b519-891d32498176" (UID: "6f5e2f42-7122-4574-b519-891d32498176"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:10.669998 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.669968 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fa61ca-48d1-4fb0-99c9-2781775881d4-plugins-config-volume" pod "04fa61ca-48d1-4fb0-99c9-2781775881d4" (UID: "04fa61ca-48d1-4fb0-99c9-2781775881d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:10.670141 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.670060 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f5e2f42-7122-4574-b519-891d32498176-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:10.672022 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.671995 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5e2f42-7122-4574-b519-891d32498176-kube-api-access-9m4zk" pod "6f5e2f42-7122-4574-b519-891d32498176" (UID: "6f5e2f42-7122-4574-b519-891d32498176"). InnerVolumeSpecName "kube-api-access-9m4zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:10.672150 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.672037 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fa61ca-48d1-4fb0-99c9-2781775881d4-kube-api-access-vgfl4" pod "04fa61ca-48d1-4fb0-99c9-2781775881d4" (UID: "04fa61ca-48d1-4fb0-99c9-2781775881d4"). InnerVolumeSpecName "kube-api-access-vgfl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:10.771787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.771681 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fa61ca-48d1-4fb0-99c9-2781775881d4-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:10.771787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.771706 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m4zk\" (UniqueName: \"kubernetes.io/projected/6f5e2f42-7122-4574-b519-891d32498176-kube-api-access-9m4zk\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:10.771787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:10.771716 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgfl4\" (UniqueName: \"kubernetes.io/projected/04fa61ca-48d1-4fb0-99c9-2781775881d4-kube-api-access-vgfl4\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:11.298187 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.298153 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk"] Oct 30 17:25:11.331642 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.331579 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw"] Oct 30 17:25:11.335538 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.335516 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" Oct 30 17:25:11.482689 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.482655 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9345c3c1-50db-406b-81b6-202cb9b75496-kube-api-access-wvdrw\" (UniqueName: \"kubernetes.io/projected/9345c3c1-50db-406b-81b6-202cb9b75496-kube-api-access-wvdrw\") pod \"9345c3c1-50db-406b-81b6-202cb9b75496\" (UID: \"9345c3c1-50db-406b-81b6-202cb9b75496\") " Oct 30 17:25:11.482904 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.482785 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9345c3c1-50db-406b-81b6-202cb9b75496-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9345c3c1-50db-406b-81b6-202cb9b75496-plugins-config-volume\") pod \"9345c3c1-50db-406b-81b6-202cb9b75496\" (UID: \"9345c3c1-50db-406b-81b6-202cb9b75496\") " Oct 30 17:25:11.483216 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.483189 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9345c3c1-50db-406b-81b6-202cb9b75496-plugins-config-volume" pod "9345c3c1-50db-406b-81b6-202cb9b75496" (UID: "9345c3c1-50db-406b-81b6-202cb9b75496"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:11.483351 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.483333 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9345c3c1-50db-406b-81b6-202cb9b75496-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:11.485181 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.485143 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9345c3c1-50db-406b-81b6-202cb9b75496-kube-api-access-wvdrw" pod "9345c3c1-50db-406b-81b6-202cb9b75496" (UID: "9345c3c1-50db-406b-81b6-202cb9b75496"). InnerVolumeSpecName "kube-api-access-wvdrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:11.554746 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.544454 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" Oct 30 17:25:11.554746 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.544480 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt" event={"ID":"6f5e2f42-7122-4574-b519-891d32498176","Type":"ContainerDied","Data":"882fb668612f80c0c552b86ee4b398659a230a7820f2a7b9b80ebd53925338fd"} Oct 30 17:25:11.554746 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.544544 4861 scope.go:122] "RemoveContainer" containerID="adb282a7a2e1359c761b0a3012034b96b2958687ca4c0973967b20ae195b3177" Oct 30 17:25:11.563822 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.559242 4861 generic.go:356] "Generic (PLEG): container finished" podID="9345c3c1-50db-406b-81b6-202cb9b75496" containerID="f8ccfedb4595751530406be630724798a6bb2aca4dcf15d9769b5e00aca079ed" exitCode=0 Oct 30 17:25:11.563822 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.559453 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" event={"ID":"9345c3c1-50db-406b-81b6-202cb9b75496","Type":"ContainerDied","Data":"f8ccfedb4595751530406be630724798a6bb2aca4dcf15d9769b5e00aca079ed"} Oct 30 17:25:11.563822 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.559520 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" event={"ID":"9345c3c1-50db-406b-81b6-202cb9b75496","Type":"ContainerDied","Data":"da3bdbc9b324ce9173aaf8b46a7da3ed1fbe25156ada1c5aac1dda6a2f81067c"} Oct 30 17:25:11.563822 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.559646 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm" Oct 30 17:25:11.569954 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.569925 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" Oct 30 17:25:11.569954 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.569938 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb" event={"ID":"04fa61ca-48d1-4fb0-99c9-2781775881d4","Type":"ContainerDied","Data":"99427f1861fa597fb774538b6a012824056975facaab1f30784db2d6141d299e"} Oct 30 17:25:11.574297 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.574275 4861 scope.go:122] "RemoveContainer" containerID="f8ccfedb4595751530406be630724798a6bb2aca4dcf15d9769b5e00aca079ed" Oct 30 17:25:11.584696 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.584640 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvdrw\" (UniqueName: \"kubernetes.io/projected/9345c3c1-50db-406b-81b6-202cb9b75496-kube-api-access-wvdrw\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:11.587654 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.587632 4861 scope.go:122] "RemoveContainer" containerID="f8ccfedb4595751530406be630724798a6bb2aca4dcf15d9769b5e00aca079ed" Oct 30 17:25:11.588087 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:25:11.588060 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ccfedb4595751530406be630724798a6bb2aca4dcf15d9769b5e00aca079ed\": container with ID starting with f8ccfedb4595751530406be630724798a6bb2aca4dcf15d9769b5e00aca079ed not found: ID does not exist" containerID="f8ccfedb4595751530406be630724798a6bb2aca4dcf15d9769b5e00aca079ed" Oct 30 17:25:11.588215 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.588094 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ccfedb4595751530406be630724798a6bb2aca4dcf15d9769b5e00aca079ed"} err="failed to get container status \"f8ccfedb4595751530406be630724798a6bb2aca4dcf15d9769b5e00aca079ed\": rpc error: code = NotFound desc = could not find container \"f8ccfedb4595751530406be630724798a6bb2aca4dcf15d9769b5e00aca079ed\": container with ID starting with f8ccfedb4595751530406be630724798a6bb2aca4dcf15d9769b5e00aca079ed not found: ID does not exist" Oct 30 17:25:11.588215 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.588112 4861 scope.go:122] "RemoveContainer" containerID="b924af75adf400410178ee6f8627da9d0c286db86955f5e6a92c926b0aaa615f" Oct 30 17:25:11.644925 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.644673 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt"] Oct 30 17:25:11.650268 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.650245 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2d365c-5c9d74bd74-bn4xt"] Oct 30 17:25:11.686490 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.686458 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm"] Oct 30 17:25:11.689906 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.689883 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bbc36f-8bb95d995-md8zm"] Oct 30 17:25:11.727122 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.727100 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb"] Oct 30 17:25:11.735866 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.735842 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2d365c-667d49b795-qrtlb"] Oct 30 17:25:11.817281 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.817238 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04fa61ca-48d1-4fb0-99c9-2781775881d4" path="/var/lib/kubelet/pods/04fa61ca-48d1-4fb0-99c9-2781775881d4/volumes" Oct 30 17:25:11.817674 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.817659 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f5e2f42-7122-4574-b519-891d32498176" path="/var/lib/kubelet/pods/6f5e2f42-7122-4574-b519-891d32498176/volumes" Oct 30 17:25:11.818075 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:11.818060 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9345c3c1-50db-406b-81b6-202cb9b75496" path="/var/lib/kubelet/pods/9345c3c1-50db-406b-81b6-202cb9b75496/volumes" Oct 30 17:25:12.169341 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:12.169303 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" podUID="b56c9a33-1b9e-48dc-9ebe-d80778b20e35" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:13.400151 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.400124 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" Oct 30 17:25:13.511827 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.511797 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b56c9a33-1b9e-48dc-9ebe-d80778b20e35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56c9a33-1b9e-48dc-9ebe-d80778b20e35-plugins-config-volume\") pod \"b56c9a33-1b9e-48dc-9ebe-d80778b20e35\" (UID: \"b56c9a33-1b9e-48dc-9ebe-d80778b20e35\") " Oct 30 17:25:13.511989 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.511839 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56c9a33-1b9e-48dc-9ebe-d80778b20e35-kube-api-access-vhbpl\" (UniqueName: \"kubernetes.io/projected/b56c9a33-1b9e-48dc-9ebe-d80778b20e35-kube-api-access-vhbpl\") pod \"b56c9a33-1b9e-48dc-9ebe-d80778b20e35\" (UID: \"b56c9a33-1b9e-48dc-9ebe-d80778b20e35\") " Oct 30 17:25:13.512181 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.512156 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56c9a33-1b9e-48dc-9ebe-d80778b20e35-plugins-config-volume" pod "b56c9a33-1b9e-48dc-9ebe-d80778b20e35" (UID: "b56c9a33-1b9e-48dc-9ebe-d80778b20e35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:13.512531 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.512499 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56c9a33-1b9e-48dc-9ebe-d80778b20e35-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:13.514302 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.514278 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56c9a33-1b9e-48dc-9ebe-d80778b20e35-kube-api-access-vhbpl" pod "b56c9a33-1b9e-48dc-9ebe-d80778b20e35" (UID: "b56c9a33-1b9e-48dc-9ebe-d80778b20e35"). InnerVolumeSpecName "kube-api-access-vhbpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:13.588219 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.588188 4861 generic.go:356] "Generic (PLEG): container finished" podID="b56c9a33-1b9e-48dc-9ebe-d80778b20e35" containerID="433c4aee9dd83f4d3f9fb1c11446498cf3298240b667ac24a851257979ede425" exitCode=0 Oct 30 17:25:13.588349 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.588236 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" Oct 30 17:25:13.588349 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.588279 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" event={"ID":"b56c9a33-1b9e-48dc-9ebe-d80778b20e35","Type":"ContainerDied","Data":"433c4aee9dd83f4d3f9fb1c11446498cf3298240b667ac24a851257979ede425"} Oct 30 17:25:13.588349 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.588336 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c" event={"ID":"b56c9a33-1b9e-48dc-9ebe-d80778b20e35","Type":"ContainerDied","Data":"e8f5486d8b05d5640f54fa6876fdc1adc2528c8ada258575035dbffacbd4cabd"} Oct 30 17:25:13.588543 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.588364 4861 scope.go:122] "RemoveContainer" containerID="433c4aee9dd83f4d3f9fb1c11446498cf3298240b667ac24a851257979ede425" Oct 30 17:25:13.599574 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.599556 4861 scope.go:122] "RemoveContainer" containerID="433c4aee9dd83f4d3f9fb1c11446498cf3298240b667ac24a851257979ede425" Oct 30 17:25:13.599901 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:25:13.599880 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"433c4aee9dd83f4d3f9fb1c11446498cf3298240b667ac24a851257979ede425\": container with ID starting with 433c4aee9dd83f4d3f9fb1c11446498cf3298240b667ac24a851257979ede425 not found: ID does not exist" containerID="433c4aee9dd83f4d3f9fb1c11446498cf3298240b667ac24a851257979ede425" Oct 30 17:25:13.599981 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.599911 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433c4aee9dd83f4d3f9fb1c11446498cf3298240b667ac24a851257979ede425"} err="failed to get container status \"433c4aee9dd83f4d3f9fb1c11446498cf3298240b667ac24a851257979ede425\": rpc error: code = NotFound desc = could not find container \"433c4aee9dd83f4d3f9fb1c11446498cf3298240b667ac24a851257979ede425\": container with ID starting with 433c4aee9dd83f4d3f9fb1c11446498cf3298240b667ac24a851257979ede425 not found: ID does not exist" Oct 30 17:25:13.614323 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.614299 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhbpl\" (UniqueName: \"kubernetes.io/projected/b56c9a33-1b9e-48dc-9ebe-d80778b20e35-kube-api-access-vhbpl\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:13.645149 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.645089 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c"] Oct 30 17:25:13.649219 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.649198 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4c6596-746f499f9-nt75c"] Oct 30 17:25:13.808949 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:13.808918 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b56c9a33-1b9e-48dc-9ebe-d80778b20e35" path="/var/lib/kubelet/pods/b56c9a33-1b9e-48dc-9ebe-d80778b20e35/volumes" Oct 30 17:25:16.305749 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:16.303964 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" podUID="dda01ef9-7338-4724-ba7f-b9fe73cdfd4a" containerName="payload-processing" containerID="cri-o://4dbb8d84f95251643ff0e435d0ef8e39770586c93bfff2820a9fc2c7332b3700" gracePeriod=25 Oct 30 17:25:16.342791 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:16.336820 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" podUID="3939645c-9b16-4a51-91a2-e5f043ddcf9f" containerName="payload-pre-processing" containerID="cri-o://f85fd9bd017ef5f63d9bdc2fb0768e800fe921d8ba1305f4f8ec7086cb32a7d1" gracePeriod=25 Oct 30 17:25:16.674970 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:16.674895 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" podUID="dda01ef9-7338-4724-ba7f-b9fe73cdfd4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:20.290812 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:25:20.290773 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:25:25.663884 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:25.663847 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" podUID="3939645c-9b16-4a51-91a2-e5f043ddcf9f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:26.674307 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:26.674273 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" podUID="dda01ef9-7338-4724-ba7f-b9fe73cdfd4a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:35.664009 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:35.663968 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" podUID="3939645c-9b16-4a51-91a2-e5f043ddcf9f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:35.841437 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:35.841402 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk"] Oct 30 17:25:35.863197 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:35.863164 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc"] Oct 30 17:25:36.267700 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.267664 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2"] Oct 30 17:25:36.267944 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.267882 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" podUID="b0891610-8a0f-41f6-937b-71b46b85defc" containerName="maas-api" containerID="cri-o://6ff2ee59e8748d08e7cfd4370704ed6f81c4d276eef357bfae5b0ddf02c30ce9" gracePeriod=30 Oct 30 17:25:36.399545 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.399515 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth"] Oct 30 17:25:36.399793 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.399762 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" podUID="9cfda264-9b06-4846-8245-394666f98198" containerName="istio-proxy" containerID="cri-o://82d8f0afc7cbdd937cf275f538f534f3002d34bceb31d2f351a26ea4522bd175" gracePeriod=30 Oct 30 17:25:36.624294 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.624217 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb"] Oct 30 17:25:36.662694 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.662003 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl"] Oct 30 17:25:36.673825 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:25:36.673796 4861 logging.go:55] [core] [Channel #22374 SubChannel #22375]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.217:50052", ServerName: "10.132.0.217:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.217:50052: connect: connection refused" Oct 30 17:25:36.770632 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.770602 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 17:25:36.778576 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.778550 4861 generic.go:356] "Generic (PLEG): container finished" podID="dda01ef9-7338-4724-ba7f-b9fe73cdfd4a" containerID="4dbb8d84f95251643ff0e435d0ef8e39770586c93bfff2820a9fc2c7332b3700" exitCode=0 Oct 30 17:25:36.780844 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.778629 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" event={"ID":"dda01ef9-7338-4724-ba7f-b9fe73cdfd4a","Type":"ContainerDied","Data":"4dbb8d84f95251643ff0e435d0ef8e39770586c93bfff2820a9fc2c7332b3700"} Oct 30 17:25:36.780844 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.780561 4861 generic.go:356] "Generic (PLEG): container finished" podID="3939645c-9b16-4a51-91a2-e5f043ddcf9f" containerID="f85fd9bd017ef5f63d9bdc2fb0768e800fe921d8ba1305f4f8ec7086cb32a7d1" exitCode=0 Oct 30 17:25:36.780844 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.780630 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" event={"ID":"3939645c-9b16-4a51-91a2-e5f043ddcf9f","Type":"ContainerDied","Data":"f85fd9bd017ef5f63d9bdc2fb0768e800fe921d8ba1305f4f8ec7086cb32a7d1"} Oct 30 17:25:36.782968 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.782421 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" Oct 30 17:25:36.782968 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.782342 4861 generic.go:356] "Generic (PLEG): container finished" podID="b0891610-8a0f-41f6-937b-71b46b85defc" containerID="6ff2ee59e8748d08e7cfd4370704ed6f81c4d276eef357bfae5b0ddf02c30ce9" exitCode=0 Oct 30 17:25:36.782968 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.782589 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" event={"ID":"b0891610-8a0f-41f6-937b-71b46b85defc","Type":"ContainerDied","Data":"6ff2ee59e8748d08e7cfd4370704ed6f81c4d276eef357bfae5b0ddf02c30ce9"} Oct 30 17:25:36.782968 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.782644 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2" event={"ID":"b0891610-8a0f-41f6-937b-71b46b85defc","Type":"ContainerDied","Data":"b0b72c74c74f7ab5cc6cc3de7e9508d4bf82d4d3826ac486ede882023b701362"} Oct 30 17:25:36.782968 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.782673 4861 scope.go:122] "RemoveContainer" containerID="6ff2ee59e8748d08e7cfd4370704ed6f81c4d276eef357bfae5b0ddf02c30ce9" Oct 30 17:25:36.800499 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.800475 4861 scope.go:122] "RemoveContainer" containerID="6ff2ee59e8748d08e7cfd4370704ed6f81c4d276eef357bfae5b0ddf02c30ce9" Oct 30 17:25:36.809516 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:25:36.809479 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff2ee59e8748d08e7cfd4370704ed6f81c4d276eef357bfae5b0ddf02c30ce9\": container with ID starting with 6ff2ee59e8748d08e7cfd4370704ed6f81c4d276eef357bfae5b0ddf02c30ce9 not found: ID does not exist" containerID="6ff2ee59e8748d08e7cfd4370704ed6f81c4d276eef357bfae5b0ddf02c30ce9" Oct 30 17:25:36.809661 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.809527 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff2ee59e8748d08e7cfd4370704ed6f81c4d276eef357bfae5b0ddf02c30ce9"} err="failed to get container status \"6ff2ee59e8748d08e7cfd4370704ed6f81c4d276eef357bfae5b0ddf02c30ce9\": rpc error: code = NotFound desc = could not find container \"6ff2ee59e8748d08e7cfd4370704ed6f81c4d276eef357bfae5b0ddf02c30ce9\": container with ID starting with 6ff2ee59e8748d08e7cfd4370704ed6f81c4d276eef357bfae5b0ddf02c30ce9 not found: ID does not exist" Oct 30 17:25:36.856081 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.856018 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" Oct 30 17:25:36.869190 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.869108 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" Oct 30 17:25:36.878579 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.878554 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0891610-8a0f-41f6-937b-71b46b85defc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0891610-8a0f-41f6-937b-71b46b85defc-openshift-service-ca\") pod \"b0891610-8a0f-41f6-937b-71b46b85defc\" (UID: \"b0891610-8a0f-41f6-937b-71b46b85defc\") " Oct 30 17:25:36.878780 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.878688 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0891610-8a0f-41f6-937b-71b46b85defc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0891610-8a0f-41f6-937b-71b46b85defc-maas-api-tls\") pod \"b0891610-8a0f-41f6-937b-71b46b85defc\" (UID: \"b0891610-8a0f-41f6-937b-71b46b85defc\") " Oct 30 17:25:36.878780 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.878741 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0891610-8a0f-41f6-937b-71b46b85defc-kube-api-access-jkbxw\" (UniqueName: \"kubernetes.io/projected/b0891610-8a0f-41f6-937b-71b46b85defc-kube-api-access-jkbxw\") pod \"b0891610-8a0f-41f6-937b-71b46b85defc\" (UID: \"b0891610-8a0f-41f6-937b-71b46b85defc\") " Oct 30 17:25:36.878946 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.878868 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0891610-8a0f-41f6-937b-71b46b85defc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0891610-8a0f-41f6-937b-71b46b85defc-metrics-certs\") pod \"b0891610-8a0f-41f6-937b-71b46b85defc\" (UID: \"b0891610-8a0f-41f6-937b-71b46b85defc\") " Oct 30 17:25:36.879382 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.879328 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0891610-8a0f-41f6-937b-71b46b85defc-openshift-service-ca" pod "b0891610-8a0f-41f6-937b-71b46b85defc" (UID: "b0891610-8a0f-41f6-937b-71b46b85defc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:36.879490 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.879475 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0891610-8a0f-41f6-937b-71b46b85defc-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:36.881284 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.881258 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0891610-8a0f-41f6-937b-71b46b85defc-metrics-certs" pod "b0891610-8a0f-41f6-937b-71b46b85defc" (UID: "b0891610-8a0f-41f6-937b-71b46b85defc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:36.888661 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.888617 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0891610-8a0f-41f6-937b-71b46b85defc-maas-api-tls" pod "b0891610-8a0f-41f6-937b-71b46b85defc" (UID: "b0891610-8a0f-41f6-937b-71b46b85defc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:36.890067 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.890012 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0891610-8a0f-41f6-937b-71b46b85defc-kube-api-access-jkbxw" pod "b0891610-8a0f-41f6-937b-71b46b85defc" (UID: "b0891610-8a0f-41f6-937b-71b46b85defc"). InnerVolumeSpecName "kube-api-access-jkbxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:36.980348 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.980305 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3939645c-9b16-4a51-91a2-e5f043ddcf9f-kube-api-access-p2spf\" (UniqueName: \"kubernetes.io/projected/3939645c-9b16-4a51-91a2-e5f043ddcf9f-kube-api-access-p2spf\") pod \"3939645c-9b16-4a51-91a2-e5f043ddcf9f\" (UID: \"3939645c-9b16-4a51-91a2-e5f043ddcf9f\") " Oct 30 17:25:36.980547 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.980388 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3939645c-9b16-4a51-91a2-e5f043ddcf9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3939645c-9b16-4a51-91a2-e5f043ddcf9f-plugins-config-volume\") pod \"3939645c-9b16-4a51-91a2-e5f043ddcf9f\" (UID: \"3939645c-9b16-4a51-91a2-e5f043ddcf9f\") " Oct 30 17:25:36.980547 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.980538 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda01ef9-7338-4724-ba7f-b9fe73cdfd4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda01ef9-7338-4724-ba7f-b9fe73cdfd4a-plugins-config-volume\") pod \"dda01ef9-7338-4724-ba7f-b9fe73cdfd4a\" (UID: \"dda01ef9-7338-4724-ba7f-b9fe73cdfd4a\") " Oct 30 17:25:36.980684 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.980593 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda01ef9-7338-4724-ba7f-b9fe73cdfd4a-kube-api-access-bndzq\" (UniqueName: \"kubernetes.io/projected/dda01ef9-7338-4724-ba7f-b9fe73cdfd4a-kube-api-access-bndzq\") pod \"dda01ef9-7338-4724-ba7f-b9fe73cdfd4a\" (UID: \"dda01ef9-7338-4724-ba7f-b9fe73cdfd4a\") " Oct 30 17:25:36.981180 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.981149 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda01ef9-7338-4724-ba7f-b9fe73cdfd4a-plugins-config-volume" pod "dda01ef9-7338-4724-ba7f-b9fe73cdfd4a" (UID: "dda01ef9-7338-4724-ba7f-b9fe73cdfd4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:36.981411 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.981385 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3939645c-9b16-4a51-91a2-e5f043ddcf9f-plugins-config-volume" pod "3939645c-9b16-4a51-91a2-e5f043ddcf9f" (UID: "3939645c-9b16-4a51-91a2-e5f043ddcf9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:36.981537 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.981506 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dda01ef9-7338-4724-ba7f-b9fe73cdfd4a-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:36.981617 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.981555 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0891610-8a0f-41f6-937b-71b46b85defc-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:36.981617 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.981571 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkbxw\" (UniqueName: \"kubernetes.io/projected/b0891610-8a0f-41f6-937b-71b46b85defc-kube-api-access-jkbxw\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:36.981617 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.981585 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0891610-8a0f-41f6-937b-71b46b85defc-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:36.983824 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.983782 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda01ef9-7338-4724-ba7f-b9fe73cdfd4a-kube-api-access-bndzq" pod "dda01ef9-7338-4724-ba7f-b9fe73cdfd4a" (UID: "dda01ef9-7338-4724-ba7f-b9fe73cdfd4a"). InnerVolumeSpecName "kube-api-access-bndzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:36.984329 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:36.984300 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3939645c-9b16-4a51-91a2-e5f043ddcf9f-kube-api-access-p2spf" pod "3939645c-9b16-4a51-91a2-e5f043ddcf9f" (UID: "3939645c-9b16-4a51-91a2-e5f043ddcf9f"). InnerVolumeSpecName "kube-api-access-p2spf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:37.083387 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.083359 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2spf\" (UniqueName: \"kubernetes.io/projected/3939645c-9b16-4a51-91a2-e5f043ddcf9f-kube-api-access-p2spf\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:37.083387 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.083385 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3939645c-9b16-4a51-91a2-e5f043ddcf9f-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:37.083583 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.083396 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bndzq\" (UniqueName: \"kubernetes.io/projected/dda01ef9-7338-4724-ba7f-b9fe73cdfd4a-kube-api-access-bndzq\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:37.134941 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.134875 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2"] Oct 30 17:25:37.138789 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.138763 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-2d365c-cb5ffd747-jwsn2"] Oct 30 17:25:37.673863 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.673821 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" podUID="dda01ef9-7338-4724-ba7f-b9fe73cdfd4a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.217:50052\" within 1s: context deadline exceeded" Oct 30 17:25:37.674259 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.673904 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" Oct 30 17:25:37.791158 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.791127 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" Oct 30 17:25:37.791347 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.791154 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk" event={"ID":"dda01ef9-7338-4724-ba7f-b9fe73cdfd4a","Type":"ContainerDied","Data":"f4a6aa5df3a8bfd81636a71f8416cd60d0fe60317c0ae6fe408c0fb7ef9d7ec3"} Oct 30 17:25:37.791347 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.791220 4861 scope.go:122] "RemoveContainer" containerID="4dbb8d84f95251643ff0e435d0ef8e39770586c93bfff2820a9fc2c7332b3700" Oct 30 17:25:37.792658 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.792591 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" Oct 30 17:25:37.792658 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.792630 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw" event={"ID":"3939645c-9b16-4a51-91a2-e5f043ddcf9f","Type":"ContainerDied","Data":"ac36a353829ae2796f8eb6ca54ee7ea1db9893b8b8ef4246ee190c14ecc978da"} Oct 30 17:25:37.803198 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.803183 4861 scope.go:122] "RemoveContainer" containerID="f85fd9bd017ef5f63d9bdc2fb0768e800fe921d8ba1305f4f8ec7086cb32a7d1" Oct 30 17:25:37.810530 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.810506 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0891610-8a0f-41f6-937b-71b46b85defc" path="/var/lib/kubelet/pods/b0891610-8a0f-41f6-937b-71b46b85defc/volumes" Oct 30 17:25:37.853790 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.853768 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk"] Oct 30 17:25:37.867457 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.867431 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2589b5-7c576c886b-4xkdk"] Oct 30 17:25:37.896287 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.896231 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw"] Oct 30 17:25:37.901609 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:37.901588 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-2589b5-786b465575-mhbsw"] Oct 30 17:25:38.288037 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:38.287999 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s"] Oct 30 17:25:39.809426 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:39.809391 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3939645c-9b16-4a51-91a2-e5f043ddcf9f" path="/var/lib/kubelet/pods/3939645c-9b16-4a51-91a2-e5f043ddcf9f/volumes" Oct 30 17:25:39.809853 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:39.809831 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda01ef9-7338-4724-ba7f-b9fe73cdfd4a" path="/var/lib/kubelet/pods/dda01ef9-7338-4724-ba7f-b9fe73cdfd4a/volumes" Oct 30 17:25:40.841927 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:40.841891 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" podUID="d17e7f96-ac06-40a0-99dd-a486a538970b" containerName="payload-processing" containerID="cri-o://4f7afc4d10012b35f8d00c8dc8dbcb6cb6cc2e75e9bea1e32584bd07d302e9ff" gracePeriod=25 Oct 30 17:25:40.864295 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:40.864264 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" podUID="4ddd17b2-6142-4f07-97a6-8ac5db77cad3" containerName="payload-pre-processing" containerID="cri-o://8a4dd259e23c4398fd01999fb71efabae3886b6ab3a9d8e1ceab749092c690b9" gracePeriod=25 Oct 30 17:25:41.627758 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.625240 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" podUID="d980004b-323c-4ebf-be4e-d0356007ba80" containerName="payload-processing" containerID="cri-o://b3b23f61a23c3c18ee7db5db622be86c85b9c493aa4d1e3ad9f3f669f3bdc90d" gracePeriod=25 Oct 30 17:25:41.662485 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.662420 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" podUID="caf4ec76-a9e6-487b-a504-67adbc815d25" containerName="payload-pre-processing" containerID="cri-o://c62a23ed7ff52a23471a60af98d37483628cb6e40b811e3d464f37d9746f0ea1" gracePeriod=25 Oct 30 17:25:41.692246 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.692223 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:25:41.825390 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.825352 4861 generic.go:356] "Generic (PLEG): container finished" podID="9cfda264-9b06-4846-8245-394666f98198" containerID="82d8f0afc7cbdd937cf275f538f534f3002d34bceb31d2f351a26ea4522bd175" exitCode=0 Oct 30 17:25:41.825541 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.825397 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" Oct 30 17:25:41.825541 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.825387 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" event={"ID":"9cfda264-9b06-4846-8245-394666f98198","Type":"ContainerDied","Data":"82d8f0afc7cbdd937cf275f538f534f3002d34bceb31d2f351a26ea4522bd175"} Oct 30 17:25:41.825541 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.825504 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth" event={"ID":"9cfda264-9b06-4846-8245-394666f98198","Type":"ContainerDied","Data":"97c43ebe3f0ae54532938101ad67549e286434f705e13e3557835613c6242638"} Oct 30 17:25:41.825541 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.825529 4861 scope.go:122] "RemoveContainer" containerID="82d8f0afc7cbdd937cf275f538f534f3002d34bceb31d2f351a26ea4522bd175" Oct 30 17:25:41.838432 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.838415 4861 scope.go:122] "RemoveContainer" containerID="82d8f0afc7cbdd937cf275f538f534f3002d34bceb31d2f351a26ea4522bd175" Oct 30 17:25:41.838696 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:25:41.838673 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d8f0afc7cbdd937cf275f538f534f3002d34bceb31d2f351a26ea4522bd175\": container with ID starting with 82d8f0afc7cbdd937cf275f538f534f3002d34bceb31d2f351a26ea4522bd175 not found: ID does not exist" containerID="82d8f0afc7cbdd937cf275f538f534f3002d34bceb31d2f351a26ea4522bd175" Oct 30 17:25:41.838784 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.838704 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d8f0afc7cbdd937cf275f538f534f3002d34bceb31d2f351a26ea4522bd175"} err="failed to get container status \"82d8f0afc7cbdd937cf275f538f534f3002d34bceb31d2f351a26ea4522bd175\": rpc error: code = NotFound desc = could not find container \"82d8f0afc7cbdd937cf275f538f534f3002d34bceb31d2f351a26ea4522bd175\": container with ID starting with 82d8f0afc7cbdd937cf275f538f534f3002d34bceb31d2f351a26ea4522bd175 not found: ID does not exist" Oct 30 17:25:41.844447 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.844426 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cfda264-9b06-4846-8245-394666f98198-kube-api-access-s2hdk\" (UniqueName: \"kubernetes.io/projected/9cfda264-9b06-4846-8245-394666f98198-kube-api-access-s2hdk\") pod \"9cfda264-9b06-4846-8245-394666f98198\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " Oct 30 17:25:41.844813 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.844518 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-istio-envoy\") pod \"9cfda264-9b06-4846-8245-394666f98198\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " Oct 30 17:25:41.844813 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.844555 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-istio-data\") pod \"9cfda264-9b06-4846-8245-394666f98198\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " Oct 30 17:25:41.844813 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.844654 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cfda264-9b06-4846-8245-394666f98198-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cfda264-9b06-4846-8245-394666f98198-istio-podinfo\") pod \"9cfda264-9b06-4846-8245-394666f98198\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " Oct 30 17:25:41.844813 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.844708 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cfda264-9b06-4846-8245-394666f98198-istio-token\" (UniqueName: \"kubernetes.io/projected/9cfda264-9b06-4846-8245-394666f98198-istio-token\") pod \"9cfda264-9b06-4846-8245-394666f98198\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " Oct 30 17:25:41.844813 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.844759 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cfda264-9b06-4846-8245-394666f98198-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cfda264-9b06-4846-8245-394666f98198-istiod-ca-cert\") pod \"9cfda264-9b06-4846-8245-394666f98198\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " Oct 30 17:25:41.844813 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.844786 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-credential-socket\") pod \"9cfda264-9b06-4846-8245-394666f98198\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " Oct 30 17:25:41.845171 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.844877 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-istio-data" pod "9cfda264-9b06-4846-8245-394666f98198" (UID: "9cfda264-9b06-4846-8245-394666f98198"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:41.845171 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.845016 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-workload-socket\") pod \"9cfda264-9b06-4846-8245-394666f98198\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " Oct 30 17:25:41.845171 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.845051 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-workload-certs\") pod \"9cfda264-9b06-4846-8245-394666f98198\" (UID: \"9cfda264-9b06-4846-8245-394666f98198\") " Oct 30 17:25:41.845171 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.845069 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-credential-socket" pod "9cfda264-9b06-4846-8245-394666f98198" (UID: "9cfda264-9b06-4846-8245-394666f98198"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:41.845387 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.845181 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cfda264-9b06-4846-8245-394666f98198-istiod-ca-cert" pod "9cfda264-9b06-4846-8245-394666f98198" (UID: "9cfda264-9b06-4846-8245-394666f98198"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:41.845387 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.845265 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-workload-socket" pod "9cfda264-9b06-4846-8245-394666f98198" (UID: "9cfda264-9b06-4846-8245-394666f98198"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:41.845626 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.845604 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-workload-certs" pod "9cfda264-9b06-4846-8245-394666f98198" (UID: "9cfda264-9b06-4846-8245-394666f98198"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:41.845917 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.845661 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.845917 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.845679 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cfda264-9b06-4846-8245-394666f98198-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.845917 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.845692 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.845917 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.845703 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.846943 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.846918 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cfda264-9b06-4846-8245-394666f98198-istio-podinfo" pod "9cfda264-9b06-4846-8245-394666f98198" (UID: "9cfda264-9b06-4846-8245-394666f98198"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:25:41.847300 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.847282 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cfda264-9b06-4846-8245-394666f98198-kube-api-access-s2hdk" pod "9cfda264-9b06-4846-8245-394666f98198" (UID: "9cfda264-9b06-4846-8245-394666f98198"). InnerVolumeSpecName "kube-api-access-s2hdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:41.847396 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.847362 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cfda264-9b06-4846-8245-394666f98198-istio-token" pod "9cfda264-9b06-4846-8245-394666f98198" (UID: "9cfda264-9b06-4846-8245-394666f98198"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:41.847469 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.847450 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-istio-envoy" pod "9cfda264-9b06-4846-8245-394666f98198" (UID: "9cfda264-9b06-4846-8245-394666f98198"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:41.947329 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.947239 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cfda264-9b06-4846-8245-394666f98198-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.947329 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.947267 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.947329 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.947276 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2hdk\" (UniqueName: \"kubernetes.io/projected/9cfda264-9b06-4846-8245-394666f98198-kube-api-access-s2hdk\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.947329 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.947290 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cfda264-9b06-4846-8245-394666f98198-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.947329 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:41.947302 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cfda264-9b06-4846-8245-394666f98198-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:42.179819 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:42.179785 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth"] Oct 30 17:25:42.193282 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:42.193256 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-f94fb5-openshift-default-65bfc49749-wttth"] Oct 30 17:25:43.289487 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:43.289445 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" podUID="be98f9c8-1b20-40fa-84f5-74f69b2271c5" containerName="payload-processing" containerID="cri-o://27483c960fb63cd3bdfe2b9d46c952f8102077b667a73c5b77cbcdbe0cfb9aea" gracePeriod=25 Oct 30 17:25:43.297991 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:43.297958 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" podUID="caf4ec76-a9e6-487b-a504-67adbc815d25" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:43.298124 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:43.297958 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" podUID="d980004b-323c-4ebf-be4e-d0356007ba80" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:43.809204 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:43.809172 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cfda264-9b06-4846-8245-394666f98198" path="/var/lib/kubelet/pods/9cfda264-9b06-4846-8245-394666f98198/volumes" Oct 30 17:25:44.270420 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:44.270389 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j"] Oct 30 17:25:44.270953 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:44.270900 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" podUID="fa8794c5-5a39-47d8-b870-c7cf40526fb6" containerName="istio-proxy" containerID="cri-o://7deddd04b344f12ed4242a8df624f5df1ac72ccfac4db9629b2b06cfb556c656" gracePeriod=30 Oct 30 17:25:49.589346 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.589318 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 17:25:49.640625 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.640590 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-credential-socket\") pod \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " Oct 30 17:25:49.640810 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.640630 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-workload-socket\") pod \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " Oct 30 17:25:49.640810 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.640771 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-data\") pod \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " Oct 30 17:25:49.640941 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.640806 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-workload-certs\") pod \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " Oct 30 17:25:49.640941 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.640863 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-podinfo\") pod \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " Oct 30 17:25:49.640941 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.640875 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-credential-socket" pod "fa8794c5-5a39-47d8-b870-c7cf40526fb6" (UID: "fa8794c5-5a39-47d8-b870-c7cf40526fb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:49.641110 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.640968 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istiod-ca-cert\") pod \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " Oct 30 17:25:49.641110 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.641020 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-data" pod "fa8794c5-5a39-47d8-b870-c7cf40526fb6" (UID: "fa8794c5-5a39-47d8-b870-c7cf40526fb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:49.641110 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.641041 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-workload-socket" pod "fa8794c5-5a39-47d8-b870-c7cf40526fb6" (UID: "fa8794c5-5a39-47d8-b870-c7cf40526fb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:49.641110 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.641051 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-envoy\") pod \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " Oct 30 17:25:49.641110 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.641039 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-workload-certs" pod "fa8794c5-5a39-47d8-b870-c7cf40526fb6" (UID: "fa8794c5-5a39-47d8-b870-c7cf40526fb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:49.641404 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.641112 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-token\" (UniqueName: \"kubernetes.io/projected/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-token\") pod \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " Oct 30 17:25:49.641404 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.641169 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa8794c5-5a39-47d8-b870-c7cf40526fb6-kube-api-access-hrqxj\" (UniqueName: \"kubernetes.io/projected/fa8794c5-5a39-47d8-b870-c7cf40526fb6-kube-api-access-hrqxj\") pod \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\" (UID: \"fa8794c5-5a39-47d8-b870-c7cf40526fb6\") " Oct 30 17:25:49.641404 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.641289 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istiod-ca-cert" pod "fa8794c5-5a39-47d8-b870-c7cf40526fb6" (UID: "fa8794c5-5a39-47d8-b870-c7cf40526fb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:49.641688 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.641665 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:49.641841 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.641705 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:49.641841 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.641809 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:49.641841 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.641828 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:49.641983 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.641841 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:49.643267 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.643240 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-envoy" pod "fa8794c5-5a39-47d8-b870-c7cf40526fb6" (UID: "fa8794c5-5a39-47d8-b870-c7cf40526fb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:25:49.643479 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.643462 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8794c5-5a39-47d8-b870-c7cf40526fb6-kube-api-access-hrqxj" pod "fa8794c5-5a39-47d8-b870-c7cf40526fb6" (UID: "fa8794c5-5a39-47d8-b870-c7cf40526fb6"). InnerVolumeSpecName "kube-api-access-hrqxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:49.643798 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.643772 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-podinfo" pod "fa8794c5-5a39-47d8-b870-c7cf40526fb6" (UID: "fa8794c5-5a39-47d8-b870-c7cf40526fb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:25:49.643890 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.643796 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-token" pod "fa8794c5-5a39-47d8-b870-c7cf40526fb6" (UID: "fa8794c5-5a39-47d8-b870-c7cf40526fb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:49.743254 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.743225 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:49.743254 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.743252 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:49.743440 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.743261 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrqxj\" (UniqueName: \"kubernetes.io/projected/fa8794c5-5a39-47d8-b870-c7cf40526fb6-kube-api-access-hrqxj\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:49.743440 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.743270 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa8794c5-5a39-47d8-b870-c7cf40526fb6-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:25:49.886825 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.886789 4861 generic.go:356] "Generic (PLEG): container finished" podID="fa8794c5-5a39-47d8-b870-c7cf40526fb6" containerID="7deddd04b344f12ed4242a8df624f5df1ac72ccfac4db9629b2b06cfb556c656" exitCode=0 Oct 30 17:25:49.886965 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.886825 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" event={"ID":"fa8794c5-5a39-47d8-b870-c7cf40526fb6","Type":"ContainerDied","Data":"7deddd04b344f12ed4242a8df624f5df1ac72ccfac4db9629b2b06cfb556c656"} Oct 30 17:25:49.886965 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.886858 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" Oct 30 17:25:49.886965 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.886875 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j" event={"ID":"fa8794c5-5a39-47d8-b870-c7cf40526fb6","Type":"ContainerDied","Data":"17d101e0d071ae9397d2cf569c754548854e1cd6053e34f52782e259b2909050"} Oct 30 17:25:49.886965 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.886902 4861 scope.go:122] "RemoveContainer" containerID="7deddd04b344f12ed4242a8df624f5df1ac72ccfac4db9629b2b06cfb556c656" Oct 30 17:25:49.897106 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.897089 4861 scope.go:122] "RemoveContainer" containerID="7deddd04b344f12ed4242a8df624f5df1ac72ccfac4db9629b2b06cfb556c656" Oct 30 17:25:49.897376 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:25:49.897349 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7deddd04b344f12ed4242a8df624f5df1ac72ccfac4db9629b2b06cfb556c656\": container with ID starting with 7deddd04b344f12ed4242a8df624f5df1ac72ccfac4db9629b2b06cfb556c656 not found: ID does not exist" containerID="7deddd04b344f12ed4242a8df624f5df1ac72ccfac4db9629b2b06cfb556c656" Oct 30 17:25:49.897484 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.897382 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7deddd04b344f12ed4242a8df624f5df1ac72ccfac4db9629b2b06cfb556c656"} err="failed to get container status \"7deddd04b344f12ed4242a8df624f5df1ac72ccfac4db9629b2b06cfb556c656\": rpc error: code = NotFound desc = could not find container \"7deddd04b344f12ed4242a8df624f5df1ac72ccfac4db9629b2b06cfb556c656\": container with ID starting with 7deddd04b344f12ed4242a8df624f5df1ac72ccfac4db9629b2b06cfb556c656 not found: ID does not exist" Oct 30 17:25:49.946565 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.946545 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j"] Oct 30 17:25:49.949774 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:49.949753 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2d365c-openshift-default-7fb84dbbd9-2gx9j"] Oct 30 17:25:50.290885 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:25:50.290846 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:25:50.635514 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:50.635431 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" podUID="4ddd17b2-6142-4f07-97a6-8ac5db77cad3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:50.635916 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:50.635436 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" podUID="d17e7f96-ac06-40a0-99dd-a486a538970b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:51.718302 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:51.718269 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" podUID="be98f9c8-1b20-40fa-84f5-74f69b2271c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:51.808550 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:51.808517 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa8794c5-5a39-47d8-b870-c7cf40526fb6" path="/var/lib/kubelet/pods/fa8794c5-5a39-47d8-b870-c7cf40526fb6/volumes" Oct 30 17:25:52.846390 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:52.846357 4861 scope.go:122] "RemoveContainer" containerID="0c140559e9dd1e7a300298d9a534075458e0f694d018c00be59ad78d53fae585" Oct 30 17:25:53.298057 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:53.298023 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" podUID="caf4ec76-a9e6-487b-a504-67adbc815d25" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:25:53.298057 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:25:53.298041 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" podUID="d980004b-323c-4ebf-be4e-d0356007ba80" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:26:00.635089 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:00.635048 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" podUID="4ddd17b2-6142-4f07-97a6-8ac5db77cad3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:26:00.635089 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:00.635079 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" podUID="d17e7f96-ac06-40a0-99dd-a486a538970b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:26:00.964805 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:00.964776 4861 generic.go:356] "Generic (PLEG): container finished" podID="d17e7f96-ac06-40a0-99dd-a486a538970b" containerID="4f7afc4d10012b35f8d00c8dc8dbcb6cb6cc2e75e9bea1e32584bd07d302e9ff" exitCode=0 Oct 30 17:26:00.964908 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:00.964854 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" event={"ID":"d17e7f96-ac06-40a0-99dd-a486a538970b","Type":"ContainerDied","Data":"4f7afc4d10012b35f8d00c8dc8dbcb6cb6cc2e75e9bea1e32584bd07d302e9ff"} Oct 30 17:26:01.136919 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.136889 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" Oct 30 17:26:01.198825 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.198685 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d17e7f96-ac06-40a0-99dd-a486a538970b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17e7f96-ac06-40a0-99dd-a486a538970b-plugins-config-volume\") pod \"d17e7f96-ac06-40a0-99dd-a486a538970b\" (UID: \"d17e7f96-ac06-40a0-99dd-a486a538970b\") " Oct 30 17:26:01.198825 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.198779 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d17e7f96-ac06-40a0-99dd-a486a538970b-kube-api-access-hhtkh\" (UniqueName: \"kubernetes.io/projected/d17e7f96-ac06-40a0-99dd-a486a538970b-kube-api-access-hhtkh\") pod \"d17e7f96-ac06-40a0-99dd-a486a538970b\" (UID: \"d17e7f96-ac06-40a0-99dd-a486a538970b\") " Oct 30 17:26:01.200553 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.200519 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17e7f96-ac06-40a0-99dd-a486a538970b-plugins-config-volume" pod "d17e7f96-ac06-40a0-99dd-a486a538970b" (UID: "d17e7f96-ac06-40a0-99dd-a486a538970b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:01.218431 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.218402 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17e7f96-ac06-40a0-99dd-a486a538970b-kube-api-access-hhtkh" pod "d17e7f96-ac06-40a0-99dd-a486a538970b" (UID: "d17e7f96-ac06-40a0-99dd-a486a538970b"). InnerVolumeSpecName "kube-api-access-hhtkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:01.220586 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.220536 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" Oct 30 17:26:01.300953 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.300919 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ddd17b2-6142-4f07-97a6-8ac5db77cad3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ddd17b2-6142-4f07-97a6-8ac5db77cad3-plugins-config-volume\") pod \"4ddd17b2-6142-4f07-97a6-8ac5db77cad3\" (UID: \"4ddd17b2-6142-4f07-97a6-8ac5db77cad3\") " Oct 30 17:26:01.301132 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.301097 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ddd17b2-6142-4f07-97a6-8ac5db77cad3-kube-api-access-4cjv5\" (UniqueName: \"kubernetes.io/projected/4ddd17b2-6142-4f07-97a6-8ac5db77cad3-kube-api-access-4cjv5\") pod \"4ddd17b2-6142-4f07-97a6-8ac5db77cad3\" (UID: \"4ddd17b2-6142-4f07-97a6-8ac5db77cad3\") " Oct 30 17:26:01.301290 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.301264 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ddd17b2-6142-4f07-97a6-8ac5db77cad3-plugins-config-volume" pod "4ddd17b2-6142-4f07-97a6-8ac5db77cad3" (UID: "4ddd17b2-6142-4f07-97a6-8ac5db77cad3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:01.301553 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.301522 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ddd17b2-6142-4f07-97a6-8ac5db77cad3-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:01.301553 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.301550 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17e7f96-ac06-40a0-99dd-a486a538970b-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:01.301815 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.301598 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhtkh\" (UniqueName: \"kubernetes.io/projected/d17e7f96-ac06-40a0-99dd-a486a538970b-kube-api-access-hhtkh\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:01.303647 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.303619 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ddd17b2-6142-4f07-97a6-8ac5db77cad3-kube-api-access-4cjv5" pod "4ddd17b2-6142-4f07-97a6-8ac5db77cad3" (UID: "4ddd17b2-6142-4f07-97a6-8ac5db77cad3"). InnerVolumeSpecName "kube-api-access-4cjv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:01.402316 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.402286 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cjv5\" (UniqueName: \"kubernetes.io/projected/4ddd17b2-6142-4f07-97a6-8ac5db77cad3-kube-api-access-4cjv5\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:01.718916 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.718885 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" podUID="be98f9c8-1b20-40fa-84f5-74f69b2271c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:26:01.940422 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.940396 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" Oct 30 17:26:01.973976 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.973946 4861 generic.go:356] "Generic (PLEG): container finished" podID="d980004b-323c-4ebf-be4e-d0356007ba80" containerID="b3b23f61a23c3c18ee7db5db622be86c85b9c493aa4d1e3ad9f3f669f3bdc90d" exitCode=0 Oct 30 17:26:01.974106 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.974003 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" Oct 30 17:26:01.974106 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.974020 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" event={"ID":"d980004b-323c-4ebf-be4e-d0356007ba80","Type":"ContainerDied","Data":"b3b23f61a23c3c18ee7db5db622be86c85b9c493aa4d1e3ad9f3f669f3bdc90d"} Oct 30 17:26:01.974106 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.974073 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb" event={"ID":"d980004b-323c-4ebf-be4e-d0356007ba80","Type":"ContainerDied","Data":"94cbd0b3142dd5d31b82f9320539235be7dd81d695bfe3155205b7671115b6f3"} Oct 30 17:26:01.974106 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.974094 4861 scope.go:122] "RemoveContainer" containerID="b3b23f61a23c3c18ee7db5db622be86c85b9c493aa4d1e3ad9f3f669f3bdc90d" Oct 30 17:26:01.975545 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.975523 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" Oct 30 17:26:01.975689 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.975557 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk" event={"ID":"d17e7f96-ac06-40a0-99dd-a486a538970b","Type":"ContainerDied","Data":"52a806c4cf3bb94c233733e4adfbd657978124f9ab8bf287d6a9d4f5ed93ddde"} Oct 30 17:26:01.977139 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.977115 4861 generic.go:356] "Generic (PLEG): container finished" podID="caf4ec76-a9e6-487b-a504-67adbc815d25" containerID="c62a23ed7ff52a23471a60af98d37483628cb6e40b811e3d464f37d9746f0ea1" exitCode=0 Oct 30 17:26:01.977253 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.977177 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" event={"ID":"caf4ec76-a9e6-487b-a504-67adbc815d25","Type":"ContainerDied","Data":"c62a23ed7ff52a23471a60af98d37483628cb6e40b811e3d464f37d9746f0ea1"} Oct 30 17:26:01.977253 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.977227 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" event={"ID":"caf4ec76-a9e6-487b-a504-67adbc815d25","Type":"ContainerDied","Data":"55d890661eebee9340c476ac788886f8b58807e794b42e7781f1806075a02547"} Oct 30 17:26:01.977386 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.977252 4861 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55d890661eebee9340c476ac788886f8b58807e794b42e7781f1806075a02547" Oct 30 17:26:01.978699 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.978681 4861 generic.go:356] "Generic (PLEG): container finished" podID="4ddd17b2-6142-4f07-97a6-8ac5db77cad3" containerID="8a4dd259e23c4398fd01999fb71efabae3886b6ab3a9d8e1ceab749092c690b9" exitCode=0 Oct 30 17:26:01.978807 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.978709 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" event={"ID":"4ddd17b2-6142-4f07-97a6-8ac5db77cad3","Type":"ContainerDied","Data":"8a4dd259e23c4398fd01999fb71efabae3886b6ab3a9d8e1ceab749092c690b9"} Oct 30 17:26:01.978807 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.978766 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" event={"ID":"4ddd17b2-6142-4f07-97a6-8ac5db77cad3","Type":"ContainerDied","Data":"2b274eaae2530178acc70ed8f22e0e687705ddb84dc02a62cc99a7f388a02e46"} Oct 30 17:26:01.978807 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.978779 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc" Oct 30 17:26:01.992155 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.992110 4861 scope.go:122] "RemoveContainer" containerID="b3b23f61a23c3c18ee7db5db622be86c85b9c493aa4d1e3ad9f3f669f3bdc90d" Oct 30 17:26:01.992883 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:26:01.992769 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3b23f61a23c3c18ee7db5db622be86c85b9c493aa4d1e3ad9f3f669f3bdc90d\": container with ID starting with b3b23f61a23c3c18ee7db5db622be86c85b9c493aa4d1e3ad9f3f669f3bdc90d not found: ID does not exist" containerID="b3b23f61a23c3c18ee7db5db622be86c85b9c493aa4d1e3ad9f3f669f3bdc90d" Oct 30 17:26:01.992883 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.992805 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3b23f61a23c3c18ee7db5db622be86c85b9c493aa4d1e3ad9f3f669f3bdc90d"} err="failed to get container status \"b3b23f61a23c3c18ee7db5db622be86c85b9c493aa4d1e3ad9f3f669f3bdc90d\": rpc error: code = NotFound desc = could not find container \"b3b23f61a23c3c18ee7db5db622be86c85b9c493aa4d1e3ad9f3f669f3bdc90d\": container with ID starting with b3b23f61a23c3c18ee7db5db622be86c85b9c493aa4d1e3ad9f3f669f3bdc90d not found: ID does not exist" Oct 30 17:26:01.992883 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.992830 4861 scope.go:122] "RemoveContainer" containerID="4f7afc4d10012b35f8d00c8dc8dbcb6cb6cc2e75e9bea1e32584bd07d302e9ff" Oct 30 17:26:01.999265 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:01.999244 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" Oct 30 17:26:02.005320 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.005297 4861 scope.go:122] "RemoveContainer" containerID="8a4dd259e23c4398fd01999fb71efabae3886b6ab3a9d8e1ceab749092c690b9" Oct 30 17:26:02.010139 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.010113 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d980004b-323c-4ebf-be4e-d0356007ba80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d980004b-323c-4ebf-be4e-d0356007ba80-plugins-config-volume\") pod \"d980004b-323c-4ebf-be4e-d0356007ba80\" (UID: \"d980004b-323c-4ebf-be4e-d0356007ba80\") " Oct 30 17:26:02.010341 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.010321 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d980004b-323c-4ebf-be4e-d0356007ba80-kube-api-access-42x4m\" (UniqueName: \"kubernetes.io/projected/d980004b-323c-4ebf-be4e-d0356007ba80-kube-api-access-42x4m\") pod \"d980004b-323c-4ebf-be4e-d0356007ba80\" (UID: \"d980004b-323c-4ebf-be4e-d0356007ba80\") " Oct 30 17:26:02.010532 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.010492 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d980004b-323c-4ebf-be4e-d0356007ba80-plugins-config-volume" pod "d980004b-323c-4ebf-be4e-d0356007ba80" (UID: "d980004b-323c-4ebf-be4e-d0356007ba80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:02.010999 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.010929 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d980004b-323c-4ebf-be4e-d0356007ba80-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:02.012559 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.012531 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d980004b-323c-4ebf-be4e-d0356007ba80-kube-api-access-42x4m" pod "d980004b-323c-4ebf-be4e-d0356007ba80" (UID: "d980004b-323c-4ebf-be4e-d0356007ba80"). InnerVolumeSpecName "kube-api-access-42x4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:02.015637 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.015615 4861 scope.go:122] "RemoveContainer" containerID="8a4dd259e23c4398fd01999fb71efabae3886b6ab3a9d8e1ceab749092c690b9" Oct 30 17:26:02.016002 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:26:02.015976 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4dd259e23c4398fd01999fb71efabae3886b6ab3a9d8e1ceab749092c690b9\": container with ID starting with 8a4dd259e23c4398fd01999fb71efabae3886b6ab3a9d8e1ceab749092c690b9 not found: ID does not exist" containerID="8a4dd259e23c4398fd01999fb71efabae3886b6ab3a9d8e1ceab749092c690b9" Oct 30 17:26:02.016105 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.016014 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4dd259e23c4398fd01999fb71efabae3886b6ab3a9d8e1ceab749092c690b9"} err="failed to get container status \"8a4dd259e23c4398fd01999fb71efabae3886b6ab3a9d8e1ceab749092c690b9\": rpc error: code = NotFound desc = could not find container \"8a4dd259e23c4398fd01999fb71efabae3886b6ab3a9d8e1ceab749092c690b9\": container with ID starting with 8a4dd259e23c4398fd01999fb71efabae3886b6ab3a9d8e1ceab749092c690b9 not found: ID does not exist" Oct 30 17:26:02.076034 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.076000 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk"] Oct 30 17:26:02.079176 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.079155 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-9bb5b1-dffd59778-jhbqk"] Oct 30 17:26:02.111633 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.111605 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf4ec76-a9e6-487b-a504-67adbc815d25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf4ec76-a9e6-487b-a504-67adbc815d25-plugins-config-volume\") pod \"caf4ec76-a9e6-487b-a504-67adbc815d25\" (UID: \"caf4ec76-a9e6-487b-a504-67adbc815d25\") " Oct 30 17:26:02.111862 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.111844 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf4ec76-a9e6-487b-a504-67adbc815d25-kube-api-access-lv7rm\" (UniqueName: \"kubernetes.io/projected/caf4ec76-a9e6-487b-a504-67adbc815d25-kube-api-access-lv7rm\") pod \"caf4ec76-a9e6-487b-a504-67adbc815d25\" (UID: \"caf4ec76-a9e6-487b-a504-67adbc815d25\") " Oct 30 17:26:02.111989 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.111962 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf4ec76-a9e6-487b-a504-67adbc815d25-plugins-config-volume" pod "caf4ec76-a9e6-487b-a504-67adbc815d25" (UID: "caf4ec76-a9e6-487b-a504-67adbc815d25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:02.112485 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.112457 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf4ec76-a9e6-487b-a504-67adbc815d25-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:02.112577 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.112488 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42x4m\" (UniqueName: \"kubernetes.io/projected/d980004b-323c-4ebf-be4e-d0356007ba80-kube-api-access-42x4m\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:02.114450 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.114414 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf4ec76-a9e6-487b-a504-67adbc815d25-kube-api-access-lv7rm" pod "caf4ec76-a9e6-487b-a504-67adbc815d25" (UID: "caf4ec76-a9e6-487b-a504-67adbc815d25"). InnerVolumeSpecName "kube-api-access-lv7rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:02.115455 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.115431 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc"] Oct 30 17:26:02.119557 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.119538 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-9bb5b1-79c75c698c-zzhdc"] Oct 30 17:26:02.213302 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.213223 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv7rm\" (UniqueName: \"kubernetes.io/projected/caf4ec76-a9e6-487b-a504-67adbc815d25-kube-api-access-lv7rm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:02.324280 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.324248 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb"] Oct 30 17:26:02.329891 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.329868 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5c6b3c-5bf77ccd4b-9vtzb"] Oct 30 17:26:02.626457 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.626374 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf"] Oct 30 17:26:02.626791 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.626735 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" podUID="63f181e5-e3d0-42e1-a509-b92d3658a9b5" containerName="maas-api" containerID="cri-o://9d938e0f89e530d2421a639a6b61afd5f2dcf2450ecbcdf098d4aa1313262a34" gracePeriod=30 Oct 30 17:26:02.939040 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.939012 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:26:02.987966 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.987934 4861 generic.go:356] "Generic (PLEG): container finished" podID="63f181e5-e3d0-42e1-a509-b92d3658a9b5" containerID="9d938e0f89e530d2421a639a6b61afd5f2dcf2450ecbcdf098d4aa1313262a34" exitCode=0 Oct 30 17:26:02.988126 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.987987 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" Oct 30 17:26:02.988126 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.988025 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" event={"ID":"63f181e5-e3d0-42e1-a509-b92d3658a9b5","Type":"ContainerDied","Data":"9d938e0f89e530d2421a639a6b61afd5f2dcf2450ecbcdf098d4aa1313262a34"} Oct 30 17:26:02.988126 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.988076 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf" event={"ID":"63f181e5-e3d0-42e1-a509-b92d3658a9b5","Type":"ContainerDied","Data":"7a3609e8dfc078d903c6be4d58ced0f1e0f2836da27da12884c1613f17a5c1ba"} Oct 30 17:26:02.988126 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:02.988097 4861 scope.go:122] "RemoveContainer" containerID="9d938e0f89e530d2421a639a6b61afd5f2dcf2450ecbcdf098d4aa1313262a34" Oct 30 17:26:03.012423 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.012395 4861 scope.go:122] "RemoveContainer" containerID="9d938e0f89e530d2421a639a6b61afd5f2dcf2450ecbcdf098d4aa1313262a34" Oct 30 17:26:03.013334 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.013310 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl" Oct 30 17:26:03.014499 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:26:03.014468 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d938e0f89e530d2421a639a6b61afd5f2dcf2450ecbcdf098d4aa1313262a34\": container with ID starting with 9d938e0f89e530d2421a639a6b61afd5f2dcf2450ecbcdf098d4aa1313262a34 not found: ID does not exist" containerID="9d938e0f89e530d2421a639a6b61afd5f2dcf2450ecbcdf098d4aa1313262a34" Oct 30 17:26:03.014596 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.014525 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d938e0f89e530d2421a639a6b61afd5f2dcf2450ecbcdf098d4aa1313262a34"} err="failed to get container status \"9d938e0f89e530d2421a639a6b61afd5f2dcf2450ecbcdf098d4aa1313262a34\": rpc error: code = NotFound desc = could not find container \"9d938e0f89e530d2421a639a6b61afd5f2dcf2450ecbcdf098d4aa1313262a34\": container with ID starting with 9d938e0f89e530d2421a639a6b61afd5f2dcf2450ecbcdf098d4aa1313262a34 not found: ID does not exist" Oct 30 17:26:03.025299 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.025278 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f181e5-e3d0-42e1-a509-b92d3658a9b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f181e5-e3d0-42e1-a509-b92d3658a9b5-openshift-service-ca\") pod \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\" (UID: \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\") " Oct 30 17:26:03.025418 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.025312 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63f181e5-e3d0-42e1-a509-b92d3658a9b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f181e5-e3d0-42e1-a509-b92d3658a9b5-metrics-certs\") pod \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\" (UID: \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\") " Oct 30 17:26:03.025484 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.025470 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f181e5-e3d0-42e1-a509-b92d3658a9b5-kube-api-access-rxrds\" (UniqueName: \"kubernetes.io/projected/63f181e5-e3d0-42e1-a509-b92d3658a9b5-kube-api-access-rxrds\") pod \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\" (UID: \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\") " Oct 30 17:26:03.025549 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.025506 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63f181e5-e3d0-42e1-a509-b92d3658a9b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63f181e5-e3d0-42e1-a509-b92d3658a9b5-maas-api-tls\") pod \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\" (UID: \"63f181e5-e3d0-42e1-a509-b92d3658a9b5\") " Oct 30 17:26:03.029004 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.028966 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f181e5-e3d0-42e1-a509-b92d3658a9b5-openshift-service-ca" pod "63f181e5-e3d0-42e1-a509-b92d3658a9b5" (UID: "63f181e5-e3d0-42e1-a509-b92d3658a9b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:03.029474 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.029449 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f181e5-e3d0-42e1-a509-b92d3658a9b5-maas-api-tls" pod "63f181e5-e3d0-42e1-a509-b92d3658a9b5" (UID: "63f181e5-e3d0-42e1-a509-b92d3658a9b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:03.038889 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.038859 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f181e5-e3d0-42e1-a509-b92d3658a9b5-kube-api-access-rxrds" pod "63f181e5-e3d0-42e1-a509-b92d3658a9b5" (UID: "63f181e5-e3d0-42e1-a509-b92d3658a9b5"). InnerVolumeSpecName "kube-api-access-rxrds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:03.040775 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.040717 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f181e5-e3d0-42e1-a509-b92d3658a9b5-metrics-certs" pod "63f181e5-e3d0-42e1-a509-b92d3658a9b5" (UID: "63f181e5-e3d0-42e1-a509-b92d3658a9b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:03.072513 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.072484 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl"] Oct 30 17:26:03.076358 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.076334 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5c6b3c-5fbb8cd59-xk2rl"] Oct 30 17:26:03.129784 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.127079 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxrds\" (UniqueName: \"kubernetes.io/projected/63f181e5-e3d0-42e1-a509-b92d3658a9b5-kube-api-access-rxrds\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:03.129784 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.127117 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63f181e5-e3d0-42e1-a509-b92d3658a9b5-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:03.129784 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.127132 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f181e5-e3d0-42e1-a509-b92d3658a9b5-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:03.129784 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.127148 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f181e5-e3d0-42e1-a509-b92d3658a9b5-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:03.338585 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.338560 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf"] Oct 30 17:26:03.342793 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.342768 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-9bb5b1-f9df9c9df-d4txf"] Oct 30 17:26:03.553462 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.553438 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" Oct 30 17:26:03.632983 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.632949 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be98f9c8-1b20-40fa-84f5-74f69b2271c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be98f9c8-1b20-40fa-84f5-74f69b2271c5-plugins-config-volume\") pod \"be98f9c8-1b20-40fa-84f5-74f69b2271c5\" (UID: \"be98f9c8-1b20-40fa-84f5-74f69b2271c5\") " Oct 30 17:26:03.633161 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.633044 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be98f9c8-1b20-40fa-84f5-74f69b2271c5-kube-api-access-vwg56\" (UniqueName: \"kubernetes.io/projected/be98f9c8-1b20-40fa-84f5-74f69b2271c5-kube-api-access-vwg56\") pod \"be98f9c8-1b20-40fa-84f5-74f69b2271c5\" (UID: \"be98f9c8-1b20-40fa-84f5-74f69b2271c5\") " Oct 30 17:26:03.633438 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.633407 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be98f9c8-1b20-40fa-84f5-74f69b2271c5-plugins-config-volume" pod "be98f9c8-1b20-40fa-84f5-74f69b2271c5" (UID: "be98f9c8-1b20-40fa-84f5-74f69b2271c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:03.633645 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.633623 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be98f9c8-1b20-40fa-84f5-74f69b2271c5-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:03.635446 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.635425 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be98f9c8-1b20-40fa-84f5-74f69b2271c5-kube-api-access-vwg56" pod "be98f9c8-1b20-40fa-84f5-74f69b2271c5" (UID: "be98f9c8-1b20-40fa-84f5-74f69b2271c5"). InnerVolumeSpecName "kube-api-access-vwg56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:03.734796 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.734762 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwg56\" (UniqueName: \"kubernetes.io/projected/be98f9c8-1b20-40fa-84f5-74f69b2271c5-kube-api-access-vwg56\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:03.808423 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.808353 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ddd17b2-6142-4f07-97a6-8ac5db77cad3" path="/var/lib/kubelet/pods/4ddd17b2-6142-4f07-97a6-8ac5db77cad3/volumes" Oct 30 17:26:03.808850 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.808833 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f181e5-e3d0-42e1-a509-b92d3658a9b5" path="/var/lib/kubelet/pods/63f181e5-e3d0-42e1-a509-b92d3658a9b5/volumes" Oct 30 17:26:03.809228 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.809215 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf4ec76-a9e6-487b-a504-67adbc815d25" path="/var/lib/kubelet/pods/caf4ec76-a9e6-487b-a504-67adbc815d25/volumes" Oct 30 17:26:03.809730 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.809700 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d17e7f96-ac06-40a0-99dd-a486a538970b" path="/var/lib/kubelet/pods/d17e7f96-ac06-40a0-99dd-a486a538970b/volumes" Oct 30 17:26:03.810201 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:03.810186 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d980004b-323c-4ebf-be4e-d0356007ba80" path="/var/lib/kubelet/pods/d980004b-323c-4ebf-be4e-d0356007ba80/volumes" Oct 30 17:26:04.023498 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:04.023462 4861 generic.go:356] "Generic (PLEG): container finished" podID="be98f9c8-1b20-40fa-84f5-74f69b2271c5" containerID="27483c960fb63cd3bdfe2b9d46c952f8102077b667a73c5b77cbcdbe0cfb9aea" exitCode=0 Oct 30 17:26:04.024030 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:04.023514 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" Oct 30 17:26:04.024030 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:04.023540 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" event={"ID":"be98f9c8-1b20-40fa-84f5-74f69b2271c5","Type":"ContainerDied","Data":"27483c960fb63cd3bdfe2b9d46c952f8102077b667a73c5b77cbcdbe0cfb9aea"} Oct 30 17:26:04.024030 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:04.023592 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s" event={"ID":"be98f9c8-1b20-40fa-84f5-74f69b2271c5","Type":"ContainerDied","Data":"d70c1d98bda9e308cc104bd9b8dbb1cbe1761829310306318eaa10b3acb5da1c"} Oct 30 17:26:04.024030 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:04.023615 4861 scope.go:122] "RemoveContainer" containerID="27483c960fb63cd3bdfe2b9d46c952f8102077b667a73c5b77cbcdbe0cfb9aea" Oct 30 17:26:04.034230 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:04.034209 4861 scope.go:122] "RemoveContainer" containerID="27483c960fb63cd3bdfe2b9d46c952f8102077b667a73c5b77cbcdbe0cfb9aea" Oct 30 17:26:04.034509 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:26:04.034488 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27483c960fb63cd3bdfe2b9d46c952f8102077b667a73c5b77cbcdbe0cfb9aea\": container with ID starting with 27483c960fb63cd3bdfe2b9d46c952f8102077b667a73c5b77cbcdbe0cfb9aea not found: ID does not exist" containerID="27483c960fb63cd3bdfe2b9d46c952f8102077b667a73c5b77cbcdbe0cfb9aea" Oct 30 17:26:04.034564 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:04.034521 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27483c960fb63cd3bdfe2b9d46c952f8102077b667a73c5b77cbcdbe0cfb9aea"} err="failed to get container status \"27483c960fb63cd3bdfe2b9d46c952f8102077b667a73c5b77cbcdbe0cfb9aea\": rpc error: code = NotFound desc = could not find container \"27483c960fb63cd3bdfe2b9d46c952f8102077b667a73c5b77cbcdbe0cfb9aea\": container with ID starting with 27483c960fb63cd3bdfe2b9d46c952f8102077b667a73c5b77cbcdbe0cfb9aea not found: ID does not exist" Oct 30 17:26:04.081063 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:04.080954 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s"] Oct 30 17:26:04.094386 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:04.094360 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3106d6-86f99d44-q9h2s"] Oct 30 17:26:05.808473 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:05.808438 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be98f9c8-1b20-40fa-84f5-74f69b2271c5" path="/var/lib/kubelet/pods/be98f9c8-1b20-40fa-84f5-74f69b2271c5/volumes" Oct 30 17:26:17.617306 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:17.617269 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7"] Oct 30 17:26:17.649057 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:17.649024 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7"] Oct 30 17:26:18.390031 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:18.389996 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr"] Oct 30 17:26:20.300598 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:26:20.300554 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:26:22.617905 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:22.617833 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" podUID="0209596e-c5bc-4751-9864-0574cabf4197" containerName="payload-processing" containerID="cri-o://9aa02aae4fc57ab7de21cbb3ab5622c166d22f635047f99a55bd8a97d6fc61d2" gracePeriod=25 Oct 30 17:26:22.649399 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:22.649351 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" podUID="956234cd-9c32-48f5-853f-7646454ab761" containerName="payload-pre-processing" containerID="cri-o://fbe479ecc5cb0f01951ace6b555938febbf8cc08a215a5252ede772ecda9d6a9" gracePeriod=25 Oct 30 17:26:23.391187 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:23.391146 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" podUID="b4ce5954-4591-4f16-84d5-36f11350dffc" containerName="payload-processing" containerID="cri-o://e70aff4f8751ae00b5c3830ff5ef1bd57ebce5998f1ff9188911da4a4d08b9b3" gracePeriod=25 Oct 30 17:26:24.409332 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:24.409292 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" podUID="956234cd-9c32-48f5-853f-7646454ab761" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:26:24.409743 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:24.409479 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" podUID="0209596e-c5bc-4751-9864-0574cabf4197" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:26:30.748138 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:30.748053 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" podUID="b4ce5954-4591-4f16-84d5-36f11350dffc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:26:33.948093 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:33.948060 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf"] Oct 30 17:26:33.948546 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:33.948296 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" podUID="f0f77f41-bbe0-4095-8102-f685cc81e7e3" containerName="maas-api" containerID="cri-o://2c64a048025213f83cd6d4b8829ba4000f966b6bc82fd288681e1e601c691721" gracePeriod=30 Oct 30 17:26:34.239928 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.239891 4861 generic.go:356] "Generic (PLEG): container finished" podID="f0f77f41-bbe0-4095-8102-f685cc81e7e3" containerID="2c64a048025213f83cd6d4b8829ba4000f966b6bc82fd288681e1e601c691721" exitCode=0 Oct 30 17:26:34.240106 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.240053 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" event={"ID":"f0f77f41-bbe0-4095-8102-f685cc81e7e3","Type":"ContainerDied","Data":"2c64a048025213f83cd6d4b8829ba4000f966b6bc82fd288681e1e601c691721"} Oct 30 17:26:34.240178 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.240108 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" event={"ID":"f0f77f41-bbe0-4095-8102-f685cc81e7e3","Type":"ContainerDied","Data":"5b9a9f633101667108cb31d01843ce4924a73a08ff9e44c8611e5e4452197c6a"} Oct 30 17:26:34.240178 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.240126 4861 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b9a9f633101667108cb31d01843ce4924a73a08ff9e44c8611e5e4452197c6a" Oct 30 17:26:34.244946 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.244895 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 17:26:34.332273 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.332243 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-metrics-certs\") pod \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " Oct 30 17:26:34.332442 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.332280 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-maas-api-tls\") pod \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " Oct 30 17:26:34.332442 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.332305 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f77f41-bbe0-4095-8102-f685cc81e7e3-kube-api-access-rsn27\" (UniqueName: \"kubernetes.io/projected/f0f77f41-bbe0-4095-8102-f685cc81e7e3-kube-api-access-rsn27\") pod \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " Oct 30 17:26:34.332442 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.332428 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f77f41-bbe0-4095-8102-f685cc81e7e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0f77f41-bbe0-4095-8102-f685cc81e7e3-openshift-service-ca\") pod \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\" (UID: \"f0f77f41-bbe0-4095-8102-f685cc81e7e3\") " Oct 30 17:26:34.332915 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.332879 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f77f41-bbe0-4095-8102-f685cc81e7e3-openshift-service-ca" pod "f0f77f41-bbe0-4095-8102-f685cc81e7e3" (UID: "f0f77f41-bbe0-4095-8102-f685cc81e7e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:34.333185 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.333112 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0f77f41-bbe0-4095-8102-f685cc81e7e3-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:34.334836 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.334809 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-maas-api-tls" pod "f0f77f41-bbe0-4095-8102-f685cc81e7e3" (UID: "f0f77f41-bbe0-4095-8102-f685cc81e7e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:34.334948 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.334901 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f77f41-bbe0-4095-8102-f685cc81e7e3-kube-api-access-rsn27" pod "f0f77f41-bbe0-4095-8102-f685cc81e7e3" (UID: "f0f77f41-bbe0-4095-8102-f685cc81e7e3"). InnerVolumeSpecName "kube-api-access-rsn27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:34.335022 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.335000 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-metrics-certs" pod "f0f77f41-bbe0-4095-8102-f685cc81e7e3" (UID: "f0f77f41-bbe0-4095-8102-f685cc81e7e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:34.420281 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.420235 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" podUID="956234cd-9c32-48f5-853f-7646454ab761" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:26:34.423408 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.422886 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" podUID="0209596e-c5bc-4751-9864-0574cabf4197" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:26:34.434833 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.434676 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:34.434833 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.434707 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0f77f41-bbe0-4095-8102-f685cc81e7e3-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:34.434833 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:34.434739 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsn27\" (UniqueName: \"kubernetes.io/projected/f0f77f41-bbe0-4095-8102-f685cc81e7e3-kube-api-access-rsn27\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:35.246311 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:35.246278 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf" Oct 30 17:26:35.307885 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:35.307849 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf"] Oct 30 17:26:35.318083 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:35.318051 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cb6251-85d9468955-6p5mf"] Oct 30 17:26:35.808690 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:35.808660 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f77f41-bbe0-4095-8102-f685cc81e7e3" path="/var/lib/kubelet/pods/f0f77f41-bbe0-4095-8102-f685cc81e7e3/volumes" Oct 30 17:26:39.962029 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:39.961994 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj"] Oct 30 17:26:39.962463 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:39.962248 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" podUID="20eaed92-2c7a-4490-bc40-4efcb335b0bb" containerName="maas-api" containerID="cri-o://81fcb0c0f8f7fc070a8e9770523cab9a1f5532adb1d5eb73b025ae806a5818e0" gracePeriod=30 Oct 30 17:26:40.265694 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.265667 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:26:40.283488 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.283451 4861 generic.go:356] "Generic (PLEG): container finished" podID="20eaed92-2c7a-4490-bc40-4efcb335b0bb" containerID="81fcb0c0f8f7fc070a8e9770523cab9a1f5532adb1d5eb73b025ae806a5818e0" exitCode=0 Oct 30 17:26:40.283654 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.283506 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" Oct 30 17:26:40.283654 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.283529 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" event={"ID":"20eaed92-2c7a-4490-bc40-4efcb335b0bb","Type":"ContainerDied","Data":"81fcb0c0f8f7fc070a8e9770523cab9a1f5532adb1d5eb73b025ae806a5818e0"} Oct 30 17:26:40.283654 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.283580 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj" event={"ID":"20eaed92-2c7a-4490-bc40-4efcb335b0bb","Type":"ContainerDied","Data":"3dc1c3ba853454b03762dbcf2ef3f8ae517e85863da87a3a90b0dda44a45292a"} Oct 30 17:26:40.283654 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.283593 4861 scope.go:122] "RemoveContainer" containerID="81fcb0c0f8f7fc070a8e9770523cab9a1f5532adb1d5eb73b025ae806a5818e0" Oct 30 17:26:40.295125 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.295103 4861 scope.go:122] "RemoveContainer" containerID="81fcb0c0f8f7fc070a8e9770523cab9a1f5532adb1d5eb73b025ae806a5818e0" Oct 30 17:26:40.295399 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:26:40.295380 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81fcb0c0f8f7fc070a8e9770523cab9a1f5532adb1d5eb73b025ae806a5818e0\": container with ID starting with 81fcb0c0f8f7fc070a8e9770523cab9a1f5532adb1d5eb73b025ae806a5818e0 not found: ID does not exist" containerID="81fcb0c0f8f7fc070a8e9770523cab9a1f5532adb1d5eb73b025ae806a5818e0" Oct 30 17:26:40.295472 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.295409 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81fcb0c0f8f7fc070a8e9770523cab9a1f5532adb1d5eb73b025ae806a5818e0"} err="failed to get container status \"81fcb0c0f8f7fc070a8e9770523cab9a1f5532adb1d5eb73b025ae806a5818e0\": rpc error: code = NotFound desc = could not find container \"81fcb0c0f8f7fc070a8e9770523cab9a1f5532adb1d5eb73b025ae806a5818e0\": container with ID starting with 81fcb0c0f8f7fc070a8e9770523cab9a1f5532adb1d5eb73b025ae806a5818e0 not found: ID does not exist" Oct 30 17:26:40.409290 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.404396 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr"] Oct 30 17:26:40.409290 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.404665 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" podUID="276abb2e-f978-4d16-bfd4-c8838f173e31" containerName="maas-api" containerID="cri-o://8daeb9c38418430fc2aa949446321f086c2c9bb40741d213db60be2674fbc5bf" gracePeriod=30 Oct 30 17:26:40.409290 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.404876 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20eaed92-2c7a-4490-bc40-4efcb335b0bb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20eaed92-2c7a-4490-bc40-4efcb335b0bb-openshift-service-ca\") pod \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\" (UID: \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\") " Oct 30 17:26:40.409290 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.404944 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20eaed92-2c7a-4490-bc40-4efcb335b0bb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20eaed92-2c7a-4490-bc40-4efcb335b0bb-metrics-certs\") pod \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\" (UID: \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\") " Oct 30 17:26:40.409290 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.405019 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20eaed92-2c7a-4490-bc40-4efcb335b0bb-kube-api-access-wcsrf\" (UniqueName: \"kubernetes.io/projected/20eaed92-2c7a-4490-bc40-4efcb335b0bb-kube-api-access-wcsrf\") pod \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\" (UID: \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\") " Oct 30 17:26:40.409290 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.405078 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20eaed92-2c7a-4490-bc40-4efcb335b0bb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20eaed92-2c7a-4490-bc40-4efcb335b0bb-maas-api-tls\") pod \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\" (UID: \"20eaed92-2c7a-4490-bc40-4efcb335b0bb\") " Oct 30 17:26:40.409290 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.406377 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20eaed92-2c7a-4490-bc40-4efcb335b0bb-openshift-service-ca" pod "20eaed92-2c7a-4490-bc40-4efcb335b0bb" (UID: "20eaed92-2c7a-4490-bc40-4efcb335b0bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:40.417995 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.417939 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20eaed92-2c7a-4490-bc40-4efcb335b0bb-kube-api-access-wcsrf" pod "20eaed92-2c7a-4490-bc40-4efcb335b0bb" (UID: "20eaed92-2c7a-4490-bc40-4efcb335b0bb"). InnerVolumeSpecName "kube-api-access-wcsrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:40.424500 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.424473 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20eaed92-2c7a-4490-bc40-4efcb335b0bb-maas-api-tls" pod "20eaed92-2c7a-4490-bc40-4efcb335b0bb" (UID: "20eaed92-2c7a-4490-bc40-4efcb335b0bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:40.426921 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.426569 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20eaed92-2c7a-4490-bc40-4efcb335b0bb-metrics-certs" pod "20eaed92-2c7a-4490-bc40-4efcb335b0bb" (UID: "20eaed92-2c7a-4490-bc40-4efcb335b0bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:40.506560 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.506481 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20eaed92-2c7a-4490-bc40-4efcb335b0bb-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:40.506560 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.506514 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20eaed92-2c7a-4490-bc40-4efcb335b0bb-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:40.506560 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.506531 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcsrf\" (UniqueName: \"kubernetes.io/projected/20eaed92-2c7a-4490-bc40-4efcb335b0bb-kube-api-access-wcsrf\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:40.506560 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.506542 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20eaed92-2c7a-4490-bc40-4efcb335b0bb-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:40.637755 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.637691 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj"] Oct 30 17:26:40.647000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.646963 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-43197b-6f689b4495-hvttj"] Oct 30 17:26:40.668664 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.668641 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:26:40.709097 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.709070 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/276abb2e-f978-4d16-bfd4-c8838f173e31-metrics-certs\" (UniqueName: \"kubernetes.io/secret/276abb2e-f978-4d16-bfd4-c8838f173e31-metrics-certs\") pod \"276abb2e-f978-4d16-bfd4-c8838f173e31\" (UID: \"276abb2e-f978-4d16-bfd4-c8838f173e31\") " Oct 30 17:26:40.709239 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.709154 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/276abb2e-f978-4d16-bfd4-c8838f173e31-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/276abb2e-f978-4d16-bfd4-c8838f173e31-openshift-service-ca\") pod \"276abb2e-f978-4d16-bfd4-c8838f173e31\" (UID: \"276abb2e-f978-4d16-bfd4-c8838f173e31\") " Oct 30 17:26:40.709356 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.709339 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/276abb2e-f978-4d16-bfd4-c8838f173e31-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276abb2e-f978-4d16-bfd4-c8838f173e31-maas-api-tls\") pod \"276abb2e-f978-4d16-bfd4-c8838f173e31\" (UID: \"276abb2e-f978-4d16-bfd4-c8838f173e31\") " Oct 30 17:26:40.709453 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.709412 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/276abb2e-f978-4d16-bfd4-c8838f173e31-kube-api-access-2lvng\" (UniqueName: \"kubernetes.io/projected/276abb2e-f978-4d16-bfd4-c8838f173e31-kube-api-access-2lvng\") pod \"276abb2e-f978-4d16-bfd4-c8838f173e31\" (UID: \"276abb2e-f978-4d16-bfd4-c8838f173e31\") " Oct 30 17:26:40.709565 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.709542 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276abb2e-f978-4d16-bfd4-c8838f173e31-openshift-service-ca" pod "276abb2e-f978-4d16-bfd4-c8838f173e31" (UID: "276abb2e-f978-4d16-bfd4-c8838f173e31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:40.709857 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.709837 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/276abb2e-f978-4d16-bfd4-c8838f173e31-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:40.711426 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.711402 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276abb2e-f978-4d16-bfd4-c8838f173e31-metrics-certs" pod "276abb2e-f978-4d16-bfd4-c8838f173e31" (UID: "276abb2e-f978-4d16-bfd4-c8838f173e31"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:40.711549 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.711534 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276abb2e-f978-4d16-bfd4-c8838f173e31-kube-api-access-2lvng" pod "276abb2e-f978-4d16-bfd4-c8838f173e31" (UID: "276abb2e-f978-4d16-bfd4-c8838f173e31"). InnerVolumeSpecName "kube-api-access-2lvng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:40.711600 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.711564 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276abb2e-f978-4d16-bfd4-c8838f173e31-maas-api-tls" pod "276abb2e-f978-4d16-bfd4-c8838f173e31" (UID: "276abb2e-f978-4d16-bfd4-c8838f173e31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:26:40.748000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.747965 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" podUID="b4ce5954-4591-4f16-84d5-36f11350dffc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:26:40.810836 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.810715 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276abb2e-f978-4d16-bfd4-c8838f173e31-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:40.810836 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.810794 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lvng\" (UniqueName: \"kubernetes.io/projected/276abb2e-f978-4d16-bfd4-c8838f173e31-kube-api-access-2lvng\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:40.810836 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:40.810814 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276abb2e-f978-4d16-bfd4-c8838f173e31-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:41.293277 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:41.293243 4861 generic.go:356] "Generic (PLEG): container finished" podID="276abb2e-f978-4d16-bfd4-c8838f173e31" containerID="8daeb9c38418430fc2aa949446321f086c2c9bb40741d213db60be2674fbc5bf" exitCode=0 Oct 30 17:26:41.293808 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:41.293284 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" event={"ID":"276abb2e-f978-4d16-bfd4-c8838f173e31","Type":"ContainerDied","Data":"8daeb9c38418430fc2aa949446321f086c2c9bb40741d213db60be2674fbc5bf"} Oct 30 17:26:41.293808 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:41.293300 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" Oct 30 17:26:41.293808 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:41.293323 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr" event={"ID":"276abb2e-f978-4d16-bfd4-c8838f173e31","Type":"ContainerDied","Data":"4876a5ddd0444f0027458f2659a0f88e30858f8e56a0d6f9e4850de1c8f5aac6"} Oct 30 17:26:41.293808 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:41.293343 4861 scope.go:122] "RemoveContainer" containerID="8daeb9c38418430fc2aa949446321f086c2c9bb40741d213db60be2674fbc5bf" Oct 30 17:26:41.305551 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:41.305534 4861 scope.go:122] "RemoveContainer" containerID="8daeb9c38418430fc2aa949446321f086c2c9bb40741d213db60be2674fbc5bf" Oct 30 17:26:41.305832 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:26:41.305811 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8daeb9c38418430fc2aa949446321f086c2c9bb40741d213db60be2674fbc5bf\": container with ID starting with 8daeb9c38418430fc2aa949446321f086c2c9bb40741d213db60be2674fbc5bf not found: ID does not exist" containerID="8daeb9c38418430fc2aa949446321f086c2c9bb40741d213db60be2674fbc5bf" Oct 30 17:26:41.305885 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:41.305841 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8daeb9c38418430fc2aa949446321f086c2c9bb40741d213db60be2674fbc5bf"} err="failed to get container status \"8daeb9c38418430fc2aa949446321f086c2c9bb40741d213db60be2674fbc5bf\": rpc error: code = NotFound desc = could not find container \"8daeb9c38418430fc2aa949446321f086c2c9bb40741d213db60be2674fbc5bf\": container with ID starting with 8daeb9c38418430fc2aa949446321f086c2c9bb40741d213db60be2674fbc5bf not found: ID does not exist" Oct 30 17:26:41.400308 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:41.400270 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr"] Oct 30 17:26:41.406000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:41.405971 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5758b7-66c6b77895-7r6fr"] Oct 30 17:26:41.808670 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:41.808637 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20eaed92-2c7a-4490-bc40-4efcb335b0bb" path="/var/lib/kubelet/pods/20eaed92-2c7a-4490-bc40-4efcb335b0bb/volumes" Oct 30 17:26:41.809326 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:41.809303 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="276abb2e-f978-4d16-bfd4-c8838f173e31" path="/var/lib/kubelet/pods/276abb2e-f978-4d16-bfd4-c8838f173e31/volumes" Oct 30 17:26:42.948191 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:42.948160 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" Oct 30 17:26:42.992069 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:42.992044 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" Oct 30 17:26:43.042257 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.042223 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/956234cd-9c32-48f5-853f-7646454ab761-kube-api-access-fxbqx\" (UniqueName: \"kubernetes.io/projected/956234cd-9c32-48f5-853f-7646454ab761-kube-api-access-fxbqx\") pod \"956234cd-9c32-48f5-853f-7646454ab761\" (UID: \"956234cd-9c32-48f5-853f-7646454ab761\") " Oct 30 17:26:43.042407 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.042320 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0209596e-c5bc-4751-9864-0574cabf4197-kube-api-access-kz5s4\" (UniqueName: \"kubernetes.io/projected/0209596e-c5bc-4751-9864-0574cabf4197-kube-api-access-kz5s4\") pod \"0209596e-c5bc-4751-9864-0574cabf4197\" (UID: \"0209596e-c5bc-4751-9864-0574cabf4197\") " Oct 30 17:26:43.042407 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.042350 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/956234cd-9c32-48f5-853f-7646454ab761-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956234cd-9c32-48f5-853f-7646454ab761-plugins-config-volume\") pod \"956234cd-9c32-48f5-853f-7646454ab761\" (UID: \"956234cd-9c32-48f5-853f-7646454ab761\") " Oct 30 17:26:43.042662 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.042618 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0209596e-c5bc-4751-9864-0574cabf4197-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0209596e-c5bc-4751-9864-0574cabf4197-plugins-config-volume\") pod \"0209596e-c5bc-4751-9864-0574cabf4197\" (UID: \"0209596e-c5bc-4751-9864-0574cabf4197\") " Oct 30 17:26:43.043054 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.042980 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956234cd-9c32-48f5-853f-7646454ab761-plugins-config-volume" pod "956234cd-9c32-48f5-853f-7646454ab761" (UID: "956234cd-9c32-48f5-853f-7646454ab761"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:43.043275 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.043259 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956234cd-9c32-48f5-853f-7646454ab761-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.044286 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.044264 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0209596e-c5bc-4751-9864-0574cabf4197-plugins-config-volume" pod "0209596e-c5bc-4751-9864-0574cabf4197" (UID: "0209596e-c5bc-4751-9864-0574cabf4197"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:43.045131 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.045105 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956234cd-9c32-48f5-853f-7646454ab761-kube-api-access-fxbqx" pod "956234cd-9c32-48f5-853f-7646454ab761" (UID: "956234cd-9c32-48f5-853f-7646454ab761"). InnerVolumeSpecName "kube-api-access-fxbqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:43.045617 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.045596 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0209596e-c5bc-4751-9864-0574cabf4197-kube-api-access-kz5s4" pod "0209596e-c5bc-4751-9864-0574cabf4197" (UID: "0209596e-c5bc-4751-9864-0574cabf4197"). InnerVolumeSpecName "kube-api-access-kz5s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:43.145117 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.145040 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz5s4\" (UniqueName: \"kubernetes.io/projected/0209596e-c5bc-4751-9864-0574cabf4197-kube-api-access-kz5s4\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.145117 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.145066 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0209596e-c5bc-4751-9864-0574cabf4197-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.145117 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.145076 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxbqx\" (UniqueName: \"kubernetes.io/projected/956234cd-9c32-48f5-853f-7646454ab761-kube-api-access-fxbqx\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.309156 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.309119 4861 generic.go:356] "Generic (PLEG): container finished" podID="956234cd-9c32-48f5-853f-7646454ab761" containerID="fbe479ecc5cb0f01951ace6b555938febbf8cc08a215a5252ede772ecda9d6a9" exitCode=0 Oct 30 17:26:43.309333 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.309164 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" Oct 30 17:26:43.309333 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.309198 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" event={"ID":"956234cd-9c32-48f5-853f-7646454ab761","Type":"ContainerDied","Data":"fbe479ecc5cb0f01951ace6b555938febbf8cc08a215a5252ede772ecda9d6a9"} Oct 30 17:26:43.309333 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.309256 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7" event={"ID":"956234cd-9c32-48f5-853f-7646454ab761","Type":"ContainerDied","Data":"ac1e19c4d11ee2190d82538ab53b000d91028d5d0cc02444a7fca851dd23474c"} Oct 30 17:26:43.309333 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.309271 4861 scope.go:122] "RemoveContainer" containerID="fbe479ecc5cb0f01951ace6b555938febbf8cc08a215a5252ede772ecda9d6a9" Oct 30 17:26:43.310854 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.310833 4861 generic.go:356] "Generic (PLEG): container finished" podID="0209596e-c5bc-4751-9864-0574cabf4197" containerID="9aa02aae4fc57ab7de21cbb3ab5622c166d22f635047f99a55bd8a97d6fc61d2" exitCode=0 Oct 30 17:26:43.310973 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.310866 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" event={"ID":"0209596e-c5bc-4751-9864-0574cabf4197","Type":"ContainerDied","Data":"9aa02aae4fc57ab7de21cbb3ab5622c166d22f635047f99a55bd8a97d6fc61d2"} Oct 30 17:26:43.311038 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.310888 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" Oct 30 17:26:43.311112 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.311092 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7" event={"ID":"0209596e-c5bc-4751-9864-0574cabf4197","Type":"ContainerDied","Data":"deeb2c1691a20edd9df7739697d731b16b5227d03c5b00236e3514156de3ef63"} Oct 30 17:26:43.320836 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.320817 4861 scope.go:122] "RemoveContainer" containerID="fbe479ecc5cb0f01951ace6b555938febbf8cc08a215a5252ede772ecda9d6a9" Oct 30 17:26:43.321159 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:26:43.321136 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe479ecc5cb0f01951ace6b555938febbf8cc08a215a5252ede772ecda9d6a9\": container with ID starting with fbe479ecc5cb0f01951ace6b555938febbf8cc08a215a5252ede772ecda9d6a9 not found: ID does not exist" containerID="fbe479ecc5cb0f01951ace6b555938febbf8cc08a215a5252ede772ecda9d6a9" Oct 30 17:26:43.321210 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.321167 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe479ecc5cb0f01951ace6b555938febbf8cc08a215a5252ede772ecda9d6a9"} err="failed to get container status \"fbe479ecc5cb0f01951ace6b555938febbf8cc08a215a5252ede772ecda9d6a9\": rpc error: code = NotFound desc = could not find container \"fbe479ecc5cb0f01951ace6b555938febbf8cc08a215a5252ede772ecda9d6a9\": container with ID starting with fbe479ecc5cb0f01951ace6b555938febbf8cc08a215a5252ede772ecda9d6a9 not found: ID does not exist" Oct 30 17:26:43.321210 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.321185 4861 scope.go:122] "RemoveContainer" containerID="9aa02aae4fc57ab7de21cbb3ab5622c166d22f635047f99a55bd8a97d6fc61d2" Oct 30 17:26:43.335631 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.335209 4861 scope.go:122] "RemoveContainer" containerID="9aa02aae4fc57ab7de21cbb3ab5622c166d22f635047f99a55bd8a97d6fc61d2" Oct 30 17:26:43.335700 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:26:43.335640 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa02aae4fc57ab7de21cbb3ab5622c166d22f635047f99a55bd8a97d6fc61d2\": container with ID starting with 9aa02aae4fc57ab7de21cbb3ab5622c166d22f635047f99a55bd8a97d6fc61d2 not found: ID does not exist" containerID="9aa02aae4fc57ab7de21cbb3ab5622c166d22f635047f99a55bd8a97d6fc61d2" Oct 30 17:26:43.335700 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.335666 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa02aae4fc57ab7de21cbb3ab5622c166d22f635047f99a55bd8a97d6fc61d2"} err="failed to get container status \"9aa02aae4fc57ab7de21cbb3ab5622c166d22f635047f99a55bd8a97d6fc61d2\": rpc error: code = NotFound desc = could not find container \"9aa02aae4fc57ab7de21cbb3ab5622c166d22f635047f99a55bd8a97d6fc61d2\": container with ID starting with 9aa02aae4fc57ab7de21cbb3ab5622c166d22f635047f99a55bd8a97d6fc61d2 not found: ID does not exist" Oct 30 17:26:43.369300 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.369276 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7"] Oct 30 17:26:43.383009 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.382982 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5758b7-7878cc45d6-jwxx7"] Oct 30 17:26:43.409708 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.409684 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7"] Oct 30 17:26:43.415127 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.415105 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5758b7-6bfbddfb75-glwx7"] Oct 30 17:26:43.632482 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.632455 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" Oct 30 17:26:43.752424 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.752392 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4ce5954-4591-4f16-84d5-36f11350dffc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ce5954-4591-4f16-84d5-36f11350dffc-plugins-config-volume\") pod \"b4ce5954-4591-4f16-84d5-36f11350dffc\" (UID: \"b4ce5954-4591-4f16-84d5-36f11350dffc\") " Oct 30 17:26:43.752684 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.752654 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ce5954-4591-4f16-84d5-36f11350dffc-kube-api-access-q7bwx\" (UniqueName: \"kubernetes.io/projected/b4ce5954-4591-4f16-84d5-36f11350dffc-kube-api-access-q7bwx\") pod \"b4ce5954-4591-4f16-84d5-36f11350dffc\" (UID: \"b4ce5954-4591-4f16-84d5-36f11350dffc\") " Oct 30 17:26:43.752839 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.752813 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ce5954-4591-4f16-84d5-36f11350dffc-plugins-config-volume" pod "b4ce5954-4591-4f16-84d5-36f11350dffc" (UID: "b4ce5954-4591-4f16-84d5-36f11350dffc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:26:43.753127 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.753108 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ce5954-4591-4f16-84d5-36f11350dffc-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:43.755401 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.755374 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ce5954-4591-4f16-84d5-36f11350dffc-kube-api-access-q7bwx" pod "b4ce5954-4591-4f16-84d5-36f11350dffc" (UID: "b4ce5954-4591-4f16-84d5-36f11350dffc"). InnerVolumeSpecName "kube-api-access-q7bwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:26:43.807596 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.807565 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0209596e-c5bc-4751-9864-0574cabf4197" path="/var/lib/kubelet/pods/0209596e-c5bc-4751-9864-0574cabf4197/volumes" Oct 30 17:26:43.808021 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.808007 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="956234cd-9c32-48f5-853f-7646454ab761" path="/var/lib/kubelet/pods/956234cd-9c32-48f5-853f-7646454ab761/volumes" Oct 30 17:26:43.854666 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:43.854637 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7bwx\" (UniqueName: \"kubernetes.io/projected/b4ce5954-4591-4f16-84d5-36f11350dffc-kube-api-access-q7bwx\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:26:44.322510 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:44.322471 4861 generic.go:356] "Generic (PLEG): container finished" podID="b4ce5954-4591-4f16-84d5-36f11350dffc" containerID="e70aff4f8751ae00b5c3830ff5ef1bd57ebce5998f1ff9188911da4a4d08b9b3" exitCode=0 Oct 30 17:26:44.323041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:44.322538 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" Oct 30 17:26:44.323041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:44.322559 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" event={"ID":"b4ce5954-4591-4f16-84d5-36f11350dffc","Type":"ContainerDied","Data":"e70aff4f8751ae00b5c3830ff5ef1bd57ebce5998f1ff9188911da4a4d08b9b3"} Oct 30 17:26:44.323041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:44.322634 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr" event={"ID":"b4ce5954-4591-4f16-84d5-36f11350dffc","Type":"ContainerDied","Data":"1d5291d07c921cee5b7dec26a0a843a3ead0bb4b2a8e1198a5f6435cdf9061ae"} Oct 30 17:26:44.323041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:44.322663 4861 scope.go:122] "RemoveContainer" containerID="e70aff4f8751ae00b5c3830ff5ef1bd57ebce5998f1ff9188911da4a4d08b9b3" Oct 30 17:26:44.332842 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:44.332823 4861 scope.go:122] "RemoveContainer" containerID="e70aff4f8751ae00b5c3830ff5ef1bd57ebce5998f1ff9188911da4a4d08b9b3" Oct 30 17:26:44.333116 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:26:44.333096 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e70aff4f8751ae00b5c3830ff5ef1bd57ebce5998f1ff9188911da4a4d08b9b3\": container with ID starting with e70aff4f8751ae00b5c3830ff5ef1bd57ebce5998f1ff9188911da4a4d08b9b3 not found: ID does not exist" containerID="e70aff4f8751ae00b5c3830ff5ef1bd57ebce5998f1ff9188911da4a4d08b9b3" Oct 30 17:26:44.333181 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:44.333128 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e70aff4f8751ae00b5c3830ff5ef1bd57ebce5998f1ff9188911da4a4d08b9b3"} err="failed to get container status \"e70aff4f8751ae00b5c3830ff5ef1bd57ebce5998f1ff9188911da4a4d08b9b3\": rpc error: code = NotFound desc = could not find container \"e70aff4f8751ae00b5c3830ff5ef1bd57ebce5998f1ff9188911da4a4d08b9b3\": container with ID starting with e70aff4f8751ae00b5c3830ff5ef1bd57ebce5998f1ff9188911da4a4d08b9b3 not found: ID does not exist" Oct 30 17:26:44.379792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:44.379712 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr"] Oct 30 17:26:44.393372 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:44.393347 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-cb6251-7f468d49f4-wqbmr"] Oct 30 17:26:45.807885 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:45.807854 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ce5954-4591-4f16-84d5-36f11350dffc" path="/var/lib/kubelet/pods/b4ce5954-4591-4f16-84d5-36f11350dffc/volumes" Oct 30 17:26:48.756487 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:48.756454 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf"] Oct 30 17:26:50.290309 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:26:50.290274 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:26:52.922032 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:52.922001 4861 scope.go:122] "RemoveContainer" containerID="c62a23ed7ff52a23471a60af98d37483628cb6e40b811e3d464f37d9746f0ea1" Oct 30 17:26:52.932066 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:52.932046 4861 scope.go:122] "RemoveContainer" containerID="2c64a048025213f83cd6d4b8829ba4000f966b6bc82fd288681e1e601c691721" Oct 30 17:26:53.757319 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:53.757264 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" podUID="0682eb07-3174-4f1f-af2c-a455e9ddc740" containerName="payload-processing" containerID="cri-o://cb7dfbeacea581abc053734fda8571a19acc94a573409b4238cf58e6139deecc" gracePeriod=25 Oct 30 17:26:54.189775 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:54.189661 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp"] Oct 30 17:26:54.224007 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:54.223981 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc"] Oct 30 17:26:55.128522 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:55.128489 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27"] Oct 30 17:26:58.367671 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:58.367629 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" podUID="0682eb07-3174-4f1f-af2c-a455e9ddc740" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:26:59.190428 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:59.190388 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" podUID="59476740-3ac5-4865-a330-505ee1c8cc0a" containerName="payload-processing" containerID="cri-o://ef438c97c4f6858a81555210e3e8d03606f2e9bbdd7b8d827e9b75a1e8c4ad7c" gracePeriod=25 Oct 30 17:26:59.224987 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:59.224952 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" podUID="4b8f2426-9e85-415e-9077-cf2471ad0875" containerName="payload-pre-processing" containerID="cri-o://8a8690e3a5cbc1b293e967da4c1c07dae431e98eb0b1aa0078f9da4587de2f24" gracePeriod=25 Oct 30 17:26:59.412903 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:59.412867 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9"] Oct 30 17:26:59.413384 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:59.413094 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" podUID="6254deaf-17a7-4214-928c-2d64d6da2684" containerName="maas-api" containerID="cri-o://a79bcf3f0bc60690cf77e38a901f6ff8bd5170d064060918f1624ac02705d07e" gracePeriod=30 Oct 30 17:26:59.759167 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:59.759129 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5"] Oct 30 17:26:59.792716 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:59.792683 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz"] Oct 30 17:26:59.909285 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:59.909261 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:26:59.960675 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:59.960596 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-metrics-certs\") pod \"6254deaf-17a7-4214-928c-2d64d6da2684\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " Oct 30 17:26:59.963180 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:26:59.963148 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-metrics-certs" pod "6254deaf-17a7-4214-928c-2d64d6da2684" (UID: "6254deaf-17a7-4214-928c-2d64d6da2684"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:00.061923 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.061892 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6254deaf-17a7-4214-928c-2d64d6da2684-kube-api-access-8bpph\" (UniqueName: \"kubernetes.io/projected/6254deaf-17a7-4214-928c-2d64d6da2684-kube-api-access-8bpph\") pod \"6254deaf-17a7-4214-928c-2d64d6da2684\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " Oct 30 17:27:00.062079 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.061948 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6254deaf-17a7-4214-928c-2d64d6da2684-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6254deaf-17a7-4214-928c-2d64d6da2684-openshift-service-ca\") pod \"6254deaf-17a7-4214-928c-2d64d6da2684\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " Oct 30 17:27:00.062079 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.062032 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-maas-api-tls\") pod \"6254deaf-17a7-4214-928c-2d64d6da2684\" (UID: \"6254deaf-17a7-4214-928c-2d64d6da2684\") " Oct 30 17:27:00.062396 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.062365 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6254deaf-17a7-4214-928c-2d64d6da2684-openshift-service-ca" pod "6254deaf-17a7-4214-928c-2d64d6da2684" (UID: "6254deaf-17a7-4214-928c-2d64d6da2684"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:00.062497 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.062448 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:00.064300 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.064271 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6254deaf-17a7-4214-928c-2d64d6da2684-kube-api-access-8bpph" pod "6254deaf-17a7-4214-928c-2d64d6da2684" (UID: "6254deaf-17a7-4214-928c-2d64d6da2684"). InnerVolumeSpecName "kube-api-access-8bpph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:00.064300 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.064279 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-maas-api-tls" pod "6254deaf-17a7-4214-928c-2d64d6da2684" (UID: "6254deaf-17a7-4214-928c-2d64d6da2684"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:00.164294 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.164221 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bpph\" (UniqueName: \"kubernetes.io/projected/6254deaf-17a7-4214-928c-2d64d6da2684-kube-api-access-8bpph\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:00.164294 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.164253 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6254deaf-17a7-4214-928c-2d64d6da2684-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:00.164294 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.164267 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6254deaf-17a7-4214-928c-2d64d6da2684-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:00.433018 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.432979 4861 generic.go:356] "Generic (PLEG): container finished" podID="6254deaf-17a7-4214-928c-2d64d6da2684" containerID="a79bcf3f0bc60690cf77e38a901f6ff8bd5170d064060918f1624ac02705d07e" exitCode=0 Oct 30 17:27:00.433473 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.433031 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" Oct 30 17:27:00.433473 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.433044 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" event={"ID":"6254deaf-17a7-4214-928c-2d64d6da2684","Type":"ContainerDied","Data":"a79bcf3f0bc60690cf77e38a901f6ff8bd5170d064060918f1624ac02705d07e"} Oct 30 17:27:00.433473 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.433095 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9" event={"ID":"6254deaf-17a7-4214-928c-2d64d6da2684","Type":"ContainerDied","Data":"4c1e9263cc4a4df0322e8ab0a4147c310493ee5e4514780f9d8b0b424acdb6b1"} Oct 30 17:27:00.433473 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.433125 4861 scope.go:122] "RemoveContainer" containerID="a79bcf3f0bc60690cf77e38a901f6ff8bd5170d064060918f1624ac02705d07e" Oct 30 17:27:00.444355 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.444332 4861 scope.go:122] "RemoveContainer" containerID="a79bcf3f0bc60690cf77e38a901f6ff8bd5170d064060918f1624ac02705d07e" Oct 30 17:27:00.444638 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:00.444618 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a79bcf3f0bc60690cf77e38a901f6ff8bd5170d064060918f1624ac02705d07e\": container with ID starting with a79bcf3f0bc60690cf77e38a901f6ff8bd5170d064060918f1624ac02705d07e not found: ID does not exist" containerID="a79bcf3f0bc60690cf77e38a901f6ff8bd5170d064060918f1624ac02705d07e" Oct 30 17:27:00.444684 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.444647 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a79bcf3f0bc60690cf77e38a901f6ff8bd5170d064060918f1624ac02705d07e"} err="failed to get container status \"a79bcf3f0bc60690cf77e38a901f6ff8bd5170d064060918f1624ac02705d07e\": rpc error: code = NotFound desc = could not find container \"a79bcf3f0bc60690cf77e38a901f6ff8bd5170d064060918f1624ac02705d07e\": container with ID starting with a79bcf3f0bc60690cf77e38a901f6ff8bd5170d064060918f1624ac02705d07e not found: ID does not exist" Oct 30 17:27:00.492422 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.492397 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9"] Oct 30 17:27:00.497996 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.497969 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d80ebb-7d9b66f845-tqqm9"] Oct 30 17:27:00.868409 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:00.868379 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7"] Oct 30 17:27:01.807404 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:01.807370 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6254deaf-17a7-4214-928c-2d64d6da2684" path="/var/lib/kubelet/pods/6254deaf-17a7-4214-928c-2d64d6da2684/volumes" Oct 30 17:27:04.760501 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:04.760463 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" podUID="2f716c0d-9107-46e1-ae3c-7518ad0a62bb" containerName="payload-processing" containerID="cri-o://c66360cd5dd7bff6b9f76eb9e9ba5448f836da652d9b72b11f4c7d1651553ad1" gracePeriod=25 Oct 30 17:27:04.794024 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:04.793990 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" podUID="3272339a-14a7-4af5-90cd-8beb09ec9e40" containerName="payload-pre-processing" containerID="cri-o://1c5f807f6062045272012ec2a972404c00f339412ea27dc62d33f23020fea13a" gracePeriod=25 Oct 30 17:27:05.869517 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:05.869465 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" podUID="cb294877-b8b7-45a5-b943-fc62473cedd7" containerName="payload-processing" containerID="cri-o://3df7225512be5fca2a8f1f1a1fa033f26083e4c2759c8dce1708704cced338f3" gracePeriod=25 Oct 30 17:27:06.552419 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:06.552376 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" podUID="59476740-3ac5-4865-a330-505ee1c8cc0a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:27:06.552638 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:06.552389 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" podUID="4b8f2426-9e85-415e-9077-cf2471ad0875" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:27:06.919585 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:06.919493 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" podUID="2f716c0d-9107-46e1-ae3c-7518ad0a62bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:27:06.919585 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:06.919561 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" podUID="3272339a-14a7-4af5-90cd-8beb09ec9e40" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:27:08.368669 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:08.368628 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" podUID="0682eb07-3174-4f1f-af2c-a455e9ddc740" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:27:10.151250 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.151210 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" containerID="cri-o://2ff854f7db9f5933e2e3434b646799b88fe24bcc38ed523357c5a76c95b6e95b" gracePeriod=45 Oct 30 17:27:10.431269 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.431241 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 17:27:10.499593 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.499553 4861 generic.go:356] "Generic (PLEG): container finished" podID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerID="2ff854f7db9f5933e2e3434b646799b88fe24bcc38ed523357c5a76c95b6e95b" exitCode=0 Oct 30 17:27:10.499791 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.499607 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" Oct 30 17:27:10.499791 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.499653 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" event={"ID":"e9a7f28a-b91a-4b4b-984d-a390a3d49776","Type":"ContainerDied","Data":"2ff854f7db9f5933e2e3434b646799b88fe24bcc38ed523357c5a76c95b6e95b"} Oct 30 17:27:10.499791 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.499707 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27" event={"ID":"e9a7f28a-b91a-4b4b-984d-a390a3d49776","Type":"ContainerDied","Data":"8159f114e2ea9ba18a35081a3d73a5b25b2f74607e9ab4129dc4bf016702539b"} Oct 30 17:27:10.499791 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.499751 4861 scope.go:122] "RemoveContainer" containerID="2ff854f7db9f5933e2e3434b646799b88fe24bcc38ed523357c5a76c95b6e95b" Oct 30 17:27:10.510373 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.510348 4861 scope.go:122] "RemoveContainer" containerID="2ff854f7db9f5933e2e3434b646799b88fe24bcc38ed523357c5a76c95b6e95b" Oct 30 17:27:10.510708 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:10.510680 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ff854f7db9f5933e2e3434b646799b88fe24bcc38ed523357c5a76c95b6e95b\": container with ID starting with 2ff854f7db9f5933e2e3434b646799b88fe24bcc38ed523357c5a76c95b6e95b not found: ID does not exist" containerID="2ff854f7db9f5933e2e3434b646799b88fe24bcc38ed523357c5a76c95b6e95b" Oct 30 17:27:10.510821 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.510754 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ff854f7db9f5933e2e3434b646799b88fe24bcc38ed523357c5a76c95b6e95b"} err="failed to get container status \"2ff854f7db9f5933e2e3434b646799b88fe24bcc38ed523357c5a76c95b6e95b\": rpc error: code = NotFound desc = could not find container \"2ff854f7db9f5933e2e3434b646799b88fe24bcc38ed523357c5a76c95b6e95b\": container with ID starting with 2ff854f7db9f5933e2e3434b646799b88fe24bcc38ed523357c5a76c95b6e95b not found: ID does not exist" Oct 30 17:27:10.511143 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.511124 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-home\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-home\") pod \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " Oct 30 17:27:10.511226 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.511211 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9a7f28a-b91a-4b4b-984d-a390a3d49776-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a7f28a-b91a-4b4b-984d-a390a3d49776-tls-certs\") pod \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " Oct 30 17:27:10.511761 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.511692 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-home" pod "e9a7f28a-b91a-4b4b-984d-a390a3d49776" (UID: "e9a7f28a-b91a-4b4b-984d-a390a3d49776"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:10.511891 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.511818 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-model-cache\") pod \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " Oct 30 17:27:10.512788 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.511907 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-dshm\") pod \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " Oct 30 17:27:10.512788 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.511946 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9a7f28a-b91a-4b4b-984d-a390a3d49776-kube-api-access-l5twj\" (UniqueName: \"kubernetes.io/projected/e9a7f28a-b91a-4b4b-984d-a390a3d49776-kube-api-access-l5twj\") pod \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " Oct 30 17:27:10.512788 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.512009 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-tmp-dir\") pod \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\" (UID: \"e9a7f28a-b91a-4b4b-984d-a390a3d49776\") " Oct 30 17:27:10.512788 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.512170 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-model-cache" pod "e9a7f28a-b91a-4b4b-984d-a390a3d49776" (UID: "e9a7f28a-b91a-4b4b-984d-a390a3d49776"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:10.512788 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.512702 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-tmp-dir" pod "e9a7f28a-b91a-4b4b-984d-a390a3d49776" (UID: "e9a7f28a-b91a-4b4b-984d-a390a3d49776"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:10.513122 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.513072 4861 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:10.513122 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.513096 4861 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:10.513122 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.513108 4861 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:10.515868 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.515843 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9a7f28a-b91a-4b4b-984d-a390a3d49776-tls-certs" pod "e9a7f28a-b91a-4b4b-984d-a390a3d49776" (UID: "e9a7f28a-b91a-4b4b-984d-a390a3d49776"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:10.515975 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.515947 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a7f28a-b91a-4b4b-984d-a390a3d49776-kube-api-access-l5twj" pod "e9a7f28a-b91a-4b4b-984d-a390a3d49776" (UID: "e9a7f28a-b91a-4b4b-984d-a390a3d49776"). InnerVolumeSpecName "kube-api-access-l5twj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:10.521928 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.521903 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-dshm" pod "e9a7f28a-b91a-4b4b-984d-a390a3d49776" (UID: "e9a7f28a-b91a-4b4b-984d-a390a3d49776"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:10.613789 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.613759 4861 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a7f28a-b91a-4b4b-984d-a390a3d49776-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:10.613789 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.613787 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5twj\" (UniqueName: \"kubernetes.io/projected/e9a7f28a-b91a-4b4b-984d-a390a3d49776-kube-api-access-l5twj\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:10.613981 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.613799 4861 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a7f28a-b91a-4b4b-984d-a390a3d49776-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:10.849082 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.849051 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27"] Oct 30 17:27:10.855672 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.855649 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-420ff3/distinct-w4-420ff3-kserve-6497d7466-nhk27"] Oct 30 17:27:10.900401 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:10.900371 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" podUID="cb294877-b8b7-45a5-b943-fc62473cedd7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:27:11.807933 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:11.807899 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" path="/var/lib/kubelet/pods/e9a7f28a-b91a-4b4b-984d-a390a3d49776/volumes" Oct 30 17:27:14.057058 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.057030 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" Oct 30 17:27:14.160987 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.160799 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0682eb07-3174-4f1f-af2c-a455e9ddc740-kube-api-access-mgvhq\" (UniqueName: \"kubernetes.io/projected/0682eb07-3174-4f1f-af2c-a455e9ddc740-kube-api-access-mgvhq\") pod \"0682eb07-3174-4f1f-af2c-a455e9ddc740\" (UID: \"0682eb07-3174-4f1f-af2c-a455e9ddc740\") " Oct 30 17:27:14.161184 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.161163 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0682eb07-3174-4f1f-af2c-a455e9ddc740-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0682eb07-3174-4f1f-af2c-a455e9ddc740-plugins-config-volume\") pod \"0682eb07-3174-4f1f-af2c-a455e9ddc740\" (UID: \"0682eb07-3174-4f1f-af2c-a455e9ddc740\") " Oct 30 17:27:14.161715 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.161682 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0682eb07-3174-4f1f-af2c-a455e9ddc740-plugins-config-volume" pod "0682eb07-3174-4f1f-af2c-a455e9ddc740" (UID: "0682eb07-3174-4f1f-af2c-a455e9ddc740"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:14.162064 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.161874 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0682eb07-3174-4f1f-af2c-a455e9ddc740-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:14.163539 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.163514 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0682eb07-3174-4f1f-af2c-a455e9ddc740-kube-api-access-mgvhq" pod "0682eb07-3174-4f1f-af2c-a455e9ddc740" (UID: "0682eb07-3174-4f1f-af2c-a455e9ddc740"). InnerVolumeSpecName "kube-api-access-mgvhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:14.205704 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.205670 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t"] Oct 30 17:27:14.206039 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.206009 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" podUID="f3b08299-0d08-470e-ab89-d77ecd29a504" containerName="istio-proxy" containerID="cri-o://65360dfe3632012aff1b3847392512cf16af52f5222884b07ffa5e88f0aab0c9" gracePeriod=30 Oct 30 17:27:14.262824 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.262797 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgvhq\" (UniqueName: \"kubernetes.io/projected/0682eb07-3174-4f1f-af2c-a455e9ddc740-kube-api-access-mgvhq\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:14.526332 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.526295 4861 generic.go:356] "Generic (PLEG): container finished" podID="0682eb07-3174-4f1f-af2c-a455e9ddc740" containerID="cb7dfbeacea581abc053734fda8571a19acc94a573409b4238cf58e6139deecc" exitCode=0 Oct 30 17:27:14.526494 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.526364 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" Oct 30 17:27:14.526494 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.526364 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" event={"ID":"0682eb07-3174-4f1f-af2c-a455e9ddc740","Type":"ContainerDied","Data":"cb7dfbeacea581abc053734fda8571a19acc94a573409b4238cf58e6139deecc"} Oct 30 17:27:14.526494 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.526422 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf" event={"ID":"0682eb07-3174-4f1f-af2c-a455e9ddc740","Type":"ContainerDied","Data":"b38e63f717a0e594f024e47647d6aa3e2c6be7c3df03e4e574397db7effe3e5e"} Oct 30 17:27:14.526494 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.526450 4861 scope.go:122] "RemoveContainer" containerID="cb7dfbeacea581abc053734fda8571a19acc94a573409b4238cf58e6139deecc" Oct 30 17:27:14.537745 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.537707 4861 scope.go:122] "RemoveContainer" containerID="cb7dfbeacea581abc053734fda8571a19acc94a573409b4238cf58e6139deecc" Oct 30 17:27:14.538048 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:14.538026 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7dfbeacea581abc053734fda8571a19acc94a573409b4238cf58e6139deecc\": container with ID starting with cb7dfbeacea581abc053734fda8571a19acc94a573409b4238cf58e6139deecc not found: ID does not exist" containerID="cb7dfbeacea581abc053734fda8571a19acc94a573409b4238cf58e6139deecc" Oct 30 17:27:14.538121 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.538056 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7dfbeacea581abc053734fda8571a19acc94a573409b4238cf58e6139deecc"} err="failed to get container status \"cb7dfbeacea581abc053734fda8571a19acc94a573409b4238cf58e6139deecc\": rpc error: code = NotFound desc = could not find container \"cb7dfbeacea581abc053734fda8571a19acc94a573409b4238cf58e6139deecc\": container with ID starting with cb7dfbeacea581abc053734fda8571a19acc94a573409b4238cf58e6139deecc not found: ID does not exist" Oct 30 17:27:14.584462 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.584435 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf"] Oct 30 17:27:14.591257 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:14.591235 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d80ebb-796967d678-fvxqf"] Oct 30 17:27:15.807173 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:15.807142 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0682eb07-3174-4f1f-af2c-a455e9ddc740" path="/var/lib/kubelet/pods/0682eb07-3174-4f1f-af2c-a455e9ddc740/volumes" Oct 30 17:27:16.552441 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:16.552406 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" podUID="59476740-3ac5-4865-a330-505ee1c8cc0a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:27:16.552662 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:16.552506 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" podUID="4b8f2426-9e85-415e-9077-cf2471ad0875" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:27:16.919177 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:16.919097 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" podUID="2f716c0d-9107-46e1-ae3c-7518ad0a62bb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:27:16.919540 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:16.919096 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" podUID="3272339a-14a7-4af5-90cd-8beb09ec9e40" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:27:19.522944 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.522598 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" Oct 30 17:27:19.542782 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.542528 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 17:27:19.561055 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.561026 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" Oct 30 17:27:19.563568 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.563338 4861 generic.go:356] "Generic (PLEG): container finished" podID="4b8f2426-9e85-415e-9077-cf2471ad0875" containerID="8a8690e3a5cbc1b293e967da4c1c07dae431e98eb0b1aa0078f9da4587de2f24" exitCode=0 Oct 30 17:27:19.563568 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.563379 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" Oct 30 17:27:19.563568 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.563401 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" event={"ID":"4b8f2426-9e85-415e-9077-cf2471ad0875","Type":"ContainerDied","Data":"8a8690e3a5cbc1b293e967da4c1c07dae431e98eb0b1aa0078f9da4587de2f24"} Oct 30 17:27:19.563568 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.563455 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc" event={"ID":"4b8f2426-9e85-415e-9077-cf2471ad0875","Type":"ContainerDied","Data":"9c924c8e17c917c397cb87976a9650b21804bb94862604607bcca111f0372b15"} Oct 30 17:27:19.563568 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.563486 4861 scope.go:122] "RemoveContainer" containerID="8a8690e3a5cbc1b293e967da4c1c07dae431e98eb0b1aa0078f9da4587de2f24" Oct 30 17:27:19.565770 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.565644 4861 generic.go:356] "Generic (PLEG): container finished" podID="59476740-3ac5-4865-a330-505ee1c8cc0a" containerID="ef438c97c4f6858a81555210e3e8d03606f2e9bbdd7b8d827e9b75a1e8c4ad7c" exitCode=0 Oct 30 17:27:19.565770 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.565694 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" Oct 30 17:27:19.566030 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.565927 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" event={"ID":"59476740-3ac5-4865-a330-505ee1c8cc0a","Type":"ContainerDied","Data":"ef438c97c4f6858a81555210e3e8d03606f2e9bbdd7b8d827e9b75a1e8c4ad7c"} Oct 30 17:27:19.566030 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.565972 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp" event={"ID":"59476740-3ac5-4865-a330-505ee1c8cc0a","Type":"ContainerDied","Data":"cdcb9f68c33b7cb85e880618eb6c4daf77a850eeb428a643773fb65f2b9c9ebb"} Oct 30 17:27:19.568251 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.568228 4861 generic.go:356] "Generic (PLEG): container finished" podID="f3b08299-0d08-470e-ab89-d77ecd29a504" containerID="65360dfe3632012aff1b3847392512cf16af52f5222884b07ffa5e88f0aab0c9" exitCode=0 Oct 30 17:27:19.568356 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.568277 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" event={"ID":"f3b08299-0d08-470e-ab89-d77ecd29a504","Type":"ContainerDied","Data":"65360dfe3632012aff1b3847392512cf16af52f5222884b07ffa5e88f0aab0c9"} Oct 30 17:27:19.568356 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.568316 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" event={"ID":"f3b08299-0d08-470e-ab89-d77ecd29a504","Type":"ContainerDied","Data":"01fb66d9568b23e97b6488a069586b339f71bf1516e203e6f458f9d6372bf1fd"} Oct 30 17:27:19.568484 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.568385 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t" Oct 30 17:27:19.576081 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.576061 4861 scope.go:122] "RemoveContainer" containerID="8a8690e3a5cbc1b293e967da4c1c07dae431e98eb0b1aa0078f9da4587de2f24" Oct 30 17:27:19.576455 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:19.576429 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8690e3a5cbc1b293e967da4c1c07dae431e98eb0b1aa0078f9da4587de2f24\": container with ID starting with 8a8690e3a5cbc1b293e967da4c1c07dae431e98eb0b1aa0078f9da4587de2f24 not found: ID does not exist" containerID="8a8690e3a5cbc1b293e967da4c1c07dae431e98eb0b1aa0078f9da4587de2f24" Oct 30 17:27:19.576545 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.576465 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8690e3a5cbc1b293e967da4c1c07dae431e98eb0b1aa0078f9da4587de2f24"} err="failed to get container status \"8a8690e3a5cbc1b293e967da4c1c07dae431e98eb0b1aa0078f9da4587de2f24\": rpc error: code = NotFound desc = could not find container \"8a8690e3a5cbc1b293e967da4c1c07dae431e98eb0b1aa0078f9da4587de2f24\": container with ID starting with 8a8690e3a5cbc1b293e967da4c1c07dae431e98eb0b1aa0078f9da4587de2f24 not found: ID does not exist" Oct 30 17:27:19.576605 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.576550 4861 scope.go:122] "RemoveContainer" containerID="ef438c97c4f6858a81555210e3e8d03606f2e9bbdd7b8d827e9b75a1e8c4ad7c" Oct 30 17:27:19.587810 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.587788 4861 scope.go:122] "RemoveContainer" containerID="ef438c97c4f6858a81555210e3e8d03606f2e9bbdd7b8d827e9b75a1e8c4ad7c" Oct 30 17:27:19.588149 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:19.588121 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef438c97c4f6858a81555210e3e8d03606f2e9bbdd7b8d827e9b75a1e8c4ad7c\": container with ID starting with ef438c97c4f6858a81555210e3e8d03606f2e9bbdd7b8d827e9b75a1e8c4ad7c not found: ID does not exist" containerID="ef438c97c4f6858a81555210e3e8d03606f2e9bbdd7b8d827e9b75a1e8c4ad7c" Oct 30 17:27:19.588247 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.588158 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef438c97c4f6858a81555210e3e8d03606f2e9bbdd7b8d827e9b75a1e8c4ad7c"} err="failed to get container status \"ef438c97c4f6858a81555210e3e8d03606f2e9bbdd7b8d827e9b75a1e8c4ad7c\": rpc error: code = NotFound desc = could not find container \"ef438c97c4f6858a81555210e3e8d03606f2e9bbdd7b8d827e9b75a1e8c4ad7c\": container with ID starting with ef438c97c4f6858a81555210e3e8d03606f2e9bbdd7b8d827e9b75a1e8c4ad7c not found: ID does not exist" Oct 30 17:27:19.588247 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.588184 4861 scope.go:122] "RemoveContainer" containerID="65360dfe3632012aff1b3847392512cf16af52f5222884b07ffa5e88f0aab0c9" Oct 30 17:27:19.598069 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.598045 4861 scope.go:122] "RemoveContainer" containerID="7970daaa67cb227919c0fb172505a6e59d1b591a28a24ed13a174e95e157ac4a" Oct 30 17:27:19.609454 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.609408 4861 scope.go:122] "RemoveContainer" containerID="65360dfe3632012aff1b3847392512cf16af52f5222884b07ffa5e88f0aab0c9" Oct 30 17:27:19.610586 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:19.610077 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65360dfe3632012aff1b3847392512cf16af52f5222884b07ffa5e88f0aab0c9\": container with ID starting with 65360dfe3632012aff1b3847392512cf16af52f5222884b07ffa5e88f0aab0c9 not found: ID does not exist" containerID="65360dfe3632012aff1b3847392512cf16af52f5222884b07ffa5e88f0aab0c9" Oct 30 17:27:19.610586 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.610127 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65360dfe3632012aff1b3847392512cf16af52f5222884b07ffa5e88f0aab0c9"} err="failed to get container status \"65360dfe3632012aff1b3847392512cf16af52f5222884b07ffa5e88f0aab0c9\": rpc error: code = NotFound desc = could not find container \"65360dfe3632012aff1b3847392512cf16af52f5222884b07ffa5e88f0aab0c9\": container with ID starting with 65360dfe3632012aff1b3847392512cf16af52f5222884b07ffa5e88f0aab0c9 not found: ID does not exist" Oct 30 17:27:19.610586 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.610201 4861 scope.go:122] "RemoveContainer" containerID="7970daaa67cb227919c0fb172505a6e59d1b591a28a24ed13a174e95e157ac4a" Oct 30 17:27:19.610816 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:19.610761 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7970daaa67cb227919c0fb172505a6e59d1b591a28a24ed13a174e95e157ac4a\": container with ID starting with 7970daaa67cb227919c0fb172505a6e59d1b591a28a24ed13a174e95e157ac4a not found: ID does not exist" containerID="7970daaa67cb227919c0fb172505a6e59d1b591a28a24ed13a174e95e157ac4a" Oct 30 17:27:19.610816 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.610788 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7970daaa67cb227919c0fb172505a6e59d1b591a28a24ed13a174e95e157ac4a"} err="failed to get container status \"7970daaa67cb227919c0fb172505a6e59d1b591a28a24ed13a174e95e157ac4a\": rpc error: code = NotFound desc = could not find container \"7970daaa67cb227919c0fb172505a6e59d1b591a28a24ed13a174e95e157ac4a\": container with ID starting with 7970daaa67cb227919c0fb172505a6e59d1b591a28a24ed13a174e95e157ac4a not found: ID does not exist" Oct 30 17:27:19.628042 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.628015 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-podinfo\") pod \"f3b08299-0d08-470e-ab89-d77ecd29a504\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " Oct 30 17:27:19.628263 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.628073 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8f2426-9e85-415e-9077-cf2471ad0875-kube-api-access-77sbf\" (UniqueName: \"kubernetes.io/projected/4b8f2426-9e85-415e-9077-cf2471ad0875-kube-api-access-77sbf\") pod \"4b8f2426-9e85-415e-9077-cf2471ad0875\" (UID: \"4b8f2426-9e85-415e-9077-cf2471ad0875\") " Oct 30 17:27:19.628263 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.628099 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59476740-3ac5-4865-a330-505ee1c8cc0a-kube-api-access-md6df\" (UniqueName: \"kubernetes.io/projected/59476740-3ac5-4865-a330-505ee1c8cc0a-kube-api-access-md6df\") pod \"59476740-3ac5-4865-a330-505ee1c8cc0a\" (UID: \"59476740-3ac5-4865-a330-505ee1c8cc0a\") " Oct 30 17:27:19.628263 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.628138 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-envoy\") pod \"f3b08299-0d08-470e-ab89-d77ecd29a504\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " Oct 30 17:27:19.628263 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.628169 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-data\") pod \"f3b08299-0d08-470e-ab89-d77ecd29a504\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " Oct 30 17:27:19.628263 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.628254 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59476740-3ac5-4865-a330-505ee1c8cc0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59476740-3ac5-4865-a330-505ee1c8cc0a-plugins-config-volume\") pod \"59476740-3ac5-4865-a330-505ee1c8cc0a\" (UID: \"59476740-3ac5-4865-a330-505ee1c8cc0a\") " Oct 30 17:27:19.628533 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.628301 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-workload-socket\") pod \"f3b08299-0d08-470e-ab89-d77ecd29a504\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " Oct 30 17:27:19.628533 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.628402 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-token\" (UniqueName: \"kubernetes.io/projected/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-token\") pod \"f3b08299-0d08-470e-ab89-d77ecd29a504\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " Oct 30 17:27:19.628533 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.628438 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b08299-0d08-470e-ab89-d77ecd29a504-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b08299-0d08-470e-ab89-d77ecd29a504-istiod-ca-cert\") pod \"f3b08299-0d08-470e-ab89-d77ecd29a504\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " Oct 30 17:27:19.628533 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.628500 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8f2426-9e85-415e-9077-cf2471ad0875-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8f2426-9e85-415e-9077-cf2471ad0875-plugins-config-volume\") pod \"4b8f2426-9e85-415e-9077-cf2471ad0875\" (UID: \"4b8f2426-9e85-415e-9077-cf2471ad0875\") " Oct 30 17:27:19.628744 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.628537 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-workload-certs\") pod \"f3b08299-0d08-470e-ab89-d77ecd29a504\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " Oct 30 17:27:19.628744 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.628567 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-credential-socket\") pod \"f3b08299-0d08-470e-ab89-d77ecd29a504\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " Oct 30 17:27:19.628744 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.628611 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b08299-0d08-470e-ab89-d77ecd29a504-kube-api-access-54495\" (UniqueName: \"kubernetes.io/projected/f3b08299-0d08-470e-ab89-d77ecd29a504-kube-api-access-54495\") pod \"f3b08299-0d08-470e-ab89-d77ecd29a504\" (UID: \"f3b08299-0d08-470e-ab89-d77ecd29a504\") " Oct 30 17:27:19.630700 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.630677 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59476740-3ac5-4865-a330-505ee1c8cc0a-kube-api-access-md6df" pod "59476740-3ac5-4865-a330-505ee1c8cc0a" (UID: "59476740-3ac5-4865-a330-505ee1c8cc0a"). InnerVolumeSpecName "kube-api-access-md6df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:19.630917 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.630893 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8f2426-9e85-415e-9077-cf2471ad0875-kube-api-access-77sbf" pod "4b8f2426-9e85-415e-9077-cf2471ad0875" (UID: "4b8f2426-9e85-415e-9077-cf2471ad0875"). InnerVolumeSpecName "kube-api-access-77sbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:19.631226 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.631206 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59476740-3ac5-4865-a330-505ee1c8cc0a-plugins-config-volume" pod "59476740-3ac5-4865-a330-505ee1c8cc0a" (UID: "59476740-3ac5-4865-a330-505ee1c8cc0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:19.631432 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.631415 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-data" pod "f3b08299-0d08-470e-ab89-d77ecd29a504" (UID: "f3b08299-0d08-470e-ab89-d77ecd29a504"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:19.631498 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.631468 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-podinfo" pod "f3b08299-0d08-470e-ab89-d77ecd29a504" (UID: "f3b08299-0d08-470e-ab89-d77ecd29a504"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:27:19.631628 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.631605 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b08299-0d08-470e-ab89-d77ecd29a504-kube-api-access-54495" pod "f3b08299-0d08-470e-ab89-d77ecd29a504" (UID: "f3b08299-0d08-470e-ab89-d77ecd29a504"). InnerVolumeSpecName "kube-api-access-54495". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:19.631669 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.631615 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-workload-socket" pod "f3b08299-0d08-470e-ab89-d77ecd29a504" (UID: "f3b08299-0d08-470e-ab89-d77ecd29a504"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:19.631712 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.631699 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8f2426-9e85-415e-9077-cf2471ad0875-plugins-config-volume" pod "4b8f2426-9e85-415e-9077-cf2471ad0875" (UID: "4b8f2426-9e85-415e-9077-cf2471ad0875"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:19.631875 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.631857 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-workload-certs" pod "f3b08299-0d08-470e-ab89-d77ecd29a504" (UID: "f3b08299-0d08-470e-ab89-d77ecd29a504"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:19.631918 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.631871 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-credential-socket" pod "f3b08299-0d08-470e-ab89-d77ecd29a504" (UID: "f3b08299-0d08-470e-ab89-d77ecd29a504"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:19.632046 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.632031 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b08299-0d08-470e-ab89-d77ecd29a504-istiod-ca-cert" pod "f3b08299-0d08-470e-ab89-d77ecd29a504" (UID: "f3b08299-0d08-470e-ab89-d77ecd29a504"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:19.632838 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.632819 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-token" pod "f3b08299-0d08-470e-ab89-d77ecd29a504" (UID: "f3b08299-0d08-470e-ab89-d77ecd29a504"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:19.633860 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.633838 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-envoy" pod "f3b08299-0d08-470e-ab89-d77ecd29a504" (UID: "f3b08299-0d08-470e-ab89-d77ecd29a504"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:19.701458 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.701423 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn"] Oct 30 17:27:19.731052 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.730969 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.731052 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.731004 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77sbf\" (UniqueName: \"kubernetes.io/projected/4b8f2426-9e85-415e-9077-cf2471ad0875-kube-api-access-77sbf\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.731052 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.731018 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md6df\" (UniqueName: \"kubernetes.io/projected/59476740-3ac5-4865-a330-505ee1c8cc0a-kube-api-access-md6df\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.731052 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.731033 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.731052 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.731044 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.731356 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.731057 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59476740-3ac5-4865-a330-505ee1c8cc0a-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.731356 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.731072 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.731356 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.731083 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b08299-0d08-470e-ab89-d77ecd29a504-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.731356 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.731096 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b08299-0d08-470e-ab89-d77ecd29a504-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.731356 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.731107 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8f2426-9e85-415e-9077-cf2471ad0875-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.731356 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.731120 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.731356 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.731134 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b08299-0d08-470e-ab89-d77ecd29a504-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.731356 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.731147 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54495\" (UniqueName: \"kubernetes.io/projected/f3b08299-0d08-470e-ab89-d77ecd29a504-kube-api-access-54495\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:19.789107 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.789068 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl"] Oct 30 17:27:19.789337 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.789306 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" podUID="ef02b08b-f178-425d-86ce-0e72bdcb63a2" containerName="maas-api" containerID="cri-o://fb96e2f8d229af0498d28fe9c2bd7ed89f49a0932d5fb0af01098ded811cde00" gracePeriod=30 Oct 30 17:27:19.910033 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.910001 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc"] Oct 30 17:27:19.917281 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.917251 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-deb721-b68ff964c-c72tc"] Oct 30 17:27:19.953404 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.953369 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t"] Oct 30 17:27:19.959346 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.959320 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-deb721-openshift-default-5ff8bc64dc-p6s6t"] Oct 30 17:27:19.999154 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:19.999078 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp"] Oct 30 17:27:20.003346 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.003318 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-deb721-5fbcbcb49b-wbslp"] Oct 30 17:27:20.051138 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.051113 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:27:20.139693 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.139660 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef02b08b-f178-425d-86ce-0e72bdcb63a2-kube-api-access-v94fr\" (UniqueName: \"kubernetes.io/projected/ef02b08b-f178-425d-86ce-0e72bdcb63a2-kube-api-access-v94fr\") pod \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " Oct 30 17:27:20.139909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.139789 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-metrics-certs\") pod \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " Oct 30 17:27:20.139909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.139818 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-maas-api-tls\") pod \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " Oct 30 17:27:20.139909 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.139850 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef02b08b-f178-425d-86ce-0e72bdcb63a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef02b08b-f178-425d-86ce-0e72bdcb63a2-openshift-service-ca\") pod \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\" (UID: \"ef02b08b-f178-425d-86ce-0e72bdcb63a2\") " Oct 30 17:27:20.140293 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.140259 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef02b08b-f178-425d-86ce-0e72bdcb63a2-openshift-service-ca" pod "ef02b08b-f178-425d-86ce-0e72bdcb63a2" (UID: "ef02b08b-f178-425d-86ce-0e72bdcb63a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:20.142237 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.142212 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-metrics-certs" pod "ef02b08b-f178-425d-86ce-0e72bdcb63a2" (UID: "ef02b08b-f178-425d-86ce-0e72bdcb63a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:20.142378 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.142364 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-maas-api-tls" pod "ef02b08b-f178-425d-86ce-0e72bdcb63a2" (UID: "ef02b08b-f178-425d-86ce-0e72bdcb63a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:20.142559 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.142533 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef02b08b-f178-425d-86ce-0e72bdcb63a2-kube-api-access-v94fr" pod "ef02b08b-f178-425d-86ce-0e72bdcb63a2" (UID: "ef02b08b-f178-425d-86ce-0e72bdcb63a2"). InnerVolumeSpecName "kube-api-access-v94fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:20.242246 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.242206 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v94fr\" (UniqueName: \"kubernetes.io/projected/ef02b08b-f178-425d-86ce-0e72bdcb63a2-kube-api-access-v94fr\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:20.242246 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.242246 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:20.242483 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.242261 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef02b08b-f178-425d-86ce-0e72bdcb63a2-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:20.242483 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.242272 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef02b08b-f178-425d-86ce-0e72bdcb63a2-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:20.292293 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:20.292195 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:27:20.578523 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.578435 4861 generic.go:356] "Generic (PLEG): container finished" podID="ef02b08b-f178-425d-86ce-0e72bdcb63a2" containerID="fb96e2f8d229af0498d28fe9c2bd7ed89f49a0932d5fb0af01098ded811cde00" exitCode=0 Oct 30 17:27:20.578523 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.578483 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" Oct 30 17:27:20.578523 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.578498 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" event={"ID":"ef02b08b-f178-425d-86ce-0e72bdcb63a2","Type":"ContainerDied","Data":"fb96e2f8d229af0498d28fe9c2bd7ed89f49a0932d5fb0af01098ded811cde00"} Oct 30 17:27:20.579128 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.578541 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl" event={"ID":"ef02b08b-f178-425d-86ce-0e72bdcb63a2","Type":"ContainerDied","Data":"efc7d118dd3c9d049d866ac3413cf07d18efa843b395224684ef3e9a18f7c4f9"} Oct 30 17:27:20.579128 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.578567 4861 scope.go:122] "RemoveContainer" containerID="fb96e2f8d229af0498d28fe9c2bd7ed89f49a0932d5fb0af01098ded811cde00" Oct 30 17:27:20.589285 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.589266 4861 scope.go:122] "RemoveContainer" containerID="fb96e2f8d229af0498d28fe9c2bd7ed89f49a0932d5fb0af01098ded811cde00" Oct 30 17:27:20.589559 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:20.589539 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb96e2f8d229af0498d28fe9c2bd7ed89f49a0932d5fb0af01098ded811cde00\": container with ID starting with fb96e2f8d229af0498d28fe9c2bd7ed89f49a0932d5fb0af01098ded811cde00 not found: ID does not exist" containerID="fb96e2f8d229af0498d28fe9c2bd7ed89f49a0932d5fb0af01098ded811cde00" Oct 30 17:27:20.589675 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.589570 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb96e2f8d229af0498d28fe9c2bd7ed89f49a0932d5fb0af01098ded811cde00"} err="failed to get container status \"fb96e2f8d229af0498d28fe9c2bd7ed89f49a0932d5fb0af01098ded811cde00\": rpc error: code = NotFound desc = could not find container \"fb96e2f8d229af0498d28fe9c2bd7ed89f49a0932d5fb0af01098ded811cde00\": container with ID starting with fb96e2f8d229af0498d28fe9c2bd7ed89f49a0932d5fb0af01098ded811cde00 not found: ID does not exist" Oct 30 17:27:20.656399 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.656366 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl"] Oct 30 17:27:20.660355 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.660324 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b09d3-75bc845db5-6rgwl"] Oct 30 17:27:20.900084 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:20.900002 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" podUID="cb294877-b8b7-45a5-b943-fc62473cedd7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:27:21.807225 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:21.807191 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8f2426-9e85-415e-9077-cf2471ad0875" path="/var/lib/kubelet/pods/4b8f2426-9e85-415e-9077-cf2471ad0875/volumes" Oct 30 17:27:21.807631 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:21.807569 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59476740-3ac5-4865-a330-505ee1c8cc0a" path="/var/lib/kubelet/pods/59476740-3ac5-4865-a330-505ee1c8cc0a/volumes" Oct 30 17:27:21.808014 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:21.807996 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef02b08b-f178-425d-86ce-0e72bdcb63a2" path="/var/lib/kubelet/pods/ef02b08b-f178-425d-86ce-0e72bdcb63a2/volumes" Oct 30 17:27:21.808795 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:21.808775 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b08299-0d08-470e-ab89-d77ecd29a504" path="/var/lib/kubelet/pods/f3b08299-0d08-470e-ab89-d77ecd29a504/volumes" Oct 30 17:27:25.061890 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.061862 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" Oct 30 17:27:25.090407 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.090382 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" Oct 30 17:27:25.200861 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.200828 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f716c0d-9107-46e1-ae3c-7518ad0a62bb-kube-api-access-g85xh\" (UniqueName: \"kubernetes.io/projected/2f716c0d-9107-46e1-ae3c-7518ad0a62bb-kube-api-access-g85xh\") pod \"2f716c0d-9107-46e1-ae3c-7518ad0a62bb\" (UID: \"2f716c0d-9107-46e1-ae3c-7518ad0a62bb\") " Oct 30 17:27:25.201079 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.200920 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3272339a-14a7-4af5-90cd-8beb09ec9e40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3272339a-14a7-4af5-90cd-8beb09ec9e40-plugins-config-volume\") pod \"3272339a-14a7-4af5-90cd-8beb09ec9e40\" (UID: \"3272339a-14a7-4af5-90cd-8beb09ec9e40\") " Oct 30 17:27:25.201079 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.200952 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3272339a-14a7-4af5-90cd-8beb09ec9e40-kube-api-access-5dk2m\" (UniqueName: \"kubernetes.io/projected/3272339a-14a7-4af5-90cd-8beb09ec9e40-kube-api-access-5dk2m\") pod \"3272339a-14a7-4af5-90cd-8beb09ec9e40\" (UID: \"3272339a-14a7-4af5-90cd-8beb09ec9e40\") " Oct 30 17:27:25.201079 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.201044 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f716c0d-9107-46e1-ae3c-7518ad0a62bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f716c0d-9107-46e1-ae3c-7518ad0a62bb-plugins-config-volume\") pod \"2f716c0d-9107-46e1-ae3c-7518ad0a62bb\" (UID: \"2f716c0d-9107-46e1-ae3c-7518ad0a62bb\") " Oct 30 17:27:25.201451 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.201414 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3272339a-14a7-4af5-90cd-8beb09ec9e40-plugins-config-volume" pod "3272339a-14a7-4af5-90cd-8beb09ec9e40" (UID: "3272339a-14a7-4af5-90cd-8beb09ec9e40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:25.201451 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.201435 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f716c0d-9107-46e1-ae3c-7518ad0a62bb-plugins-config-volume" pod "2f716c0d-9107-46e1-ae3c-7518ad0a62bb" (UID: "2f716c0d-9107-46e1-ae3c-7518ad0a62bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:25.203280 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.203255 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f716c0d-9107-46e1-ae3c-7518ad0a62bb-kube-api-access-g85xh" pod "2f716c0d-9107-46e1-ae3c-7518ad0a62bb" (UID: "2f716c0d-9107-46e1-ae3c-7518ad0a62bb"). InnerVolumeSpecName "kube-api-access-g85xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:25.203392 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.203362 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3272339a-14a7-4af5-90cd-8beb09ec9e40-kube-api-access-5dk2m" pod "3272339a-14a7-4af5-90cd-8beb09ec9e40" (UID: "3272339a-14a7-4af5-90cd-8beb09ec9e40"). InnerVolumeSpecName "kube-api-access-5dk2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:25.302203 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.302116 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f716c0d-9107-46e1-ae3c-7518ad0a62bb-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:25.302203 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.302147 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g85xh\" (UniqueName: \"kubernetes.io/projected/2f716c0d-9107-46e1-ae3c-7518ad0a62bb-kube-api-access-g85xh\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:25.302203 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.302160 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3272339a-14a7-4af5-90cd-8beb09ec9e40-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:25.302203 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.302174 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dk2m\" (UniqueName: \"kubernetes.io/projected/3272339a-14a7-4af5-90cd-8beb09ec9e40-kube-api-access-5dk2m\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:25.435585 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.435543 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78"] Oct 30 17:27:25.611615 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.611516 4861 generic.go:356] "Generic (PLEG): container finished" podID="2f716c0d-9107-46e1-ae3c-7518ad0a62bb" containerID="c66360cd5dd7bff6b9f76eb9e9ba5448f836da652d9b72b11f4c7d1651553ad1" exitCode=0 Oct 30 17:27:25.611615 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.611563 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" Oct 30 17:27:25.611615 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.611592 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" event={"ID":"2f716c0d-9107-46e1-ae3c-7518ad0a62bb","Type":"ContainerDied","Data":"c66360cd5dd7bff6b9f76eb9e9ba5448f836da652d9b72b11f4c7d1651553ad1"} Oct 30 17:27:25.611946 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.611649 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5" event={"ID":"2f716c0d-9107-46e1-ae3c-7518ad0a62bb","Type":"ContainerDied","Data":"addfec97b617bb68d71df665a0f9f7936e43c4a4e30414847a10eb4d312307ac"} Oct 30 17:27:25.611946 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.611676 4861 scope.go:122] "RemoveContainer" containerID="c66360cd5dd7bff6b9f76eb9e9ba5448f836da652d9b72b11f4c7d1651553ad1" Oct 30 17:27:25.613244 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.613218 4861 generic.go:356] "Generic (PLEG): container finished" podID="3272339a-14a7-4af5-90cd-8beb09ec9e40" containerID="1c5f807f6062045272012ec2a972404c00f339412ea27dc62d33f23020fea13a" exitCode=0 Oct 30 17:27:25.613371 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.613259 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" Oct 30 17:27:25.613371 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.613313 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" event={"ID":"3272339a-14a7-4af5-90cd-8beb09ec9e40","Type":"ContainerDied","Data":"1c5f807f6062045272012ec2a972404c00f339412ea27dc62d33f23020fea13a"} Oct 30 17:27:25.613371 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.613355 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz" event={"ID":"3272339a-14a7-4af5-90cd-8beb09ec9e40","Type":"ContainerDied","Data":"fa3e0b01b22b189ca234501a83ebd0c1fb5295c11438febd1ff8771bc3b75be8"} Oct 30 17:27:25.622183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.622164 4861 scope.go:122] "RemoveContainer" containerID="c66360cd5dd7bff6b9f76eb9e9ba5448f836da652d9b72b11f4c7d1651553ad1" Oct 30 17:27:25.622481 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:25.622453 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66360cd5dd7bff6b9f76eb9e9ba5448f836da652d9b72b11f4c7d1651553ad1\": container with ID starting with c66360cd5dd7bff6b9f76eb9e9ba5448f836da652d9b72b11f4c7d1651553ad1 not found: ID does not exist" containerID="c66360cd5dd7bff6b9f76eb9e9ba5448f836da652d9b72b11f4c7d1651553ad1" Oct 30 17:27:25.622610 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.622489 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66360cd5dd7bff6b9f76eb9e9ba5448f836da652d9b72b11f4c7d1651553ad1"} err="failed to get container status \"c66360cd5dd7bff6b9f76eb9e9ba5448f836da652d9b72b11f4c7d1651553ad1\": rpc error: code = NotFound desc = could not find container \"c66360cd5dd7bff6b9f76eb9e9ba5448f836da652d9b72b11f4c7d1651553ad1\": container with ID starting with c66360cd5dd7bff6b9f76eb9e9ba5448f836da652d9b72b11f4c7d1651553ad1 not found: ID does not exist" Oct 30 17:27:25.622610 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.622514 4861 scope.go:122] "RemoveContainer" containerID="1c5f807f6062045272012ec2a972404c00f339412ea27dc62d33f23020fea13a" Oct 30 17:27:25.631492 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.631472 4861 scope.go:122] "RemoveContainer" containerID="1c5f807f6062045272012ec2a972404c00f339412ea27dc62d33f23020fea13a" Oct 30 17:27:25.631770 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:25.631748 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c5f807f6062045272012ec2a972404c00f339412ea27dc62d33f23020fea13a\": container with ID starting with 1c5f807f6062045272012ec2a972404c00f339412ea27dc62d33f23020fea13a not found: ID does not exist" containerID="1c5f807f6062045272012ec2a972404c00f339412ea27dc62d33f23020fea13a" Oct 30 17:27:25.631857 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.631778 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5f807f6062045272012ec2a972404c00f339412ea27dc62d33f23020fea13a"} err="failed to get container status \"1c5f807f6062045272012ec2a972404c00f339412ea27dc62d33f23020fea13a\": rpc error: code = NotFound desc = could not find container \"1c5f807f6062045272012ec2a972404c00f339412ea27dc62d33f23020fea13a\": container with ID starting with 1c5f807f6062045272012ec2a972404c00f339412ea27dc62d33f23020fea13a not found: ID does not exist" Oct 30 17:27:25.675986 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.675946 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5"] Oct 30 17:27:25.679647 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.679617 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-799c76-9b98444b-mk5z5"] Oct 30 17:27:25.710230 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.710200 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz"] Oct 30 17:27:25.714382 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.714357 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-799c76-5875d67d7-kfssz"] Oct 30 17:27:25.807656 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.807624 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f716c0d-9107-46e1-ae3c-7518ad0a62bb" path="/var/lib/kubelet/pods/2f716c0d-9107-46e1-ae3c-7518ad0a62bb/volumes" Oct 30 17:27:25.808082 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:25.808063 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3272339a-14a7-4af5-90cd-8beb09ec9e40" path="/var/lib/kubelet/pods/3272339a-14a7-4af5-90cd-8beb09ec9e40/volumes" Oct 30 17:27:26.167882 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:26.167854 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" Oct 30 17:27:26.312812 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:26.312772 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb294877-b8b7-45a5-b943-fc62473cedd7-kube-api-access-glrk7\" (UniqueName: \"kubernetes.io/projected/cb294877-b8b7-45a5-b943-fc62473cedd7-kube-api-access-glrk7\") pod \"cb294877-b8b7-45a5-b943-fc62473cedd7\" (UID: \"cb294877-b8b7-45a5-b943-fc62473cedd7\") " Oct 30 17:27:26.313015 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:26.312999 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb294877-b8b7-45a5-b943-fc62473cedd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb294877-b8b7-45a5-b943-fc62473cedd7-plugins-config-volume\") pod \"cb294877-b8b7-45a5-b943-fc62473cedd7\" (UID: \"cb294877-b8b7-45a5-b943-fc62473cedd7\") " Oct 30 17:27:26.313382 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:26.313355 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb294877-b8b7-45a5-b943-fc62473cedd7-plugins-config-volume" pod "cb294877-b8b7-45a5-b943-fc62473cedd7" (UID: "cb294877-b8b7-45a5-b943-fc62473cedd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:26.313687 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:26.313501 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb294877-b8b7-45a5-b943-fc62473cedd7-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:26.315282 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:26.315251 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb294877-b8b7-45a5-b943-fc62473cedd7-kube-api-access-glrk7" pod "cb294877-b8b7-45a5-b943-fc62473cedd7" (UID: "cb294877-b8b7-45a5-b943-fc62473cedd7"). InnerVolumeSpecName "kube-api-access-glrk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:26.414671 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:26.414587 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glrk7\" (UniqueName: \"kubernetes.io/projected/cb294877-b8b7-45a5-b943-fc62473cedd7-kube-api-access-glrk7\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:26.623346 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:26.623306 4861 generic.go:356] "Generic (PLEG): container finished" podID="cb294877-b8b7-45a5-b943-fc62473cedd7" containerID="3df7225512be5fca2a8f1f1a1fa033f26083e4c2759c8dce1708704cced338f3" exitCode=0 Oct 30 17:27:26.623346 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:26.623345 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" event={"ID":"cb294877-b8b7-45a5-b943-fc62473cedd7","Type":"ContainerDied","Data":"3df7225512be5fca2a8f1f1a1fa033f26083e4c2759c8dce1708704cced338f3"} Oct 30 17:27:26.623625 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:26.623360 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" Oct 30 17:27:26.623625 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:26.623386 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7" event={"ID":"cb294877-b8b7-45a5-b943-fc62473cedd7","Type":"ContainerDied","Data":"1d28a3369333bd2c9889ea1cf93be0c67a623b304a7296a346f2f44271988d68"} Oct 30 17:27:26.623625 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:26.623411 4861 scope.go:122] "RemoveContainer" containerID="3df7225512be5fca2a8f1f1a1fa033f26083e4c2759c8dce1708704cced338f3" Oct 30 17:27:26.633779 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:26.633759 4861 scope.go:122] "RemoveContainer" containerID="3df7225512be5fca2a8f1f1a1fa033f26083e4c2759c8dce1708704cced338f3" Oct 30 17:27:26.634036 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:26.634017 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df7225512be5fca2a8f1f1a1fa033f26083e4c2759c8dce1708704cced338f3\": container with ID starting with 3df7225512be5fca2a8f1f1a1fa033f26083e4c2759c8dce1708704cced338f3 not found: ID does not exist" containerID="3df7225512be5fca2a8f1f1a1fa033f26083e4c2759c8dce1708704cced338f3" Oct 30 17:27:26.634104 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:26.634043 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df7225512be5fca2a8f1f1a1fa033f26083e4c2759c8dce1708704cced338f3"} err="failed to get container status \"3df7225512be5fca2a8f1f1a1fa033f26083e4c2759c8dce1708704cced338f3\": rpc error: code = NotFound desc = could not find container \"3df7225512be5fca2a8f1f1a1fa033f26083e4c2759c8dce1708704cced338f3\": container with ID starting with 3df7225512be5fca2a8f1f1a1fa033f26083e4c2759c8dce1708704cced338f3 not found: ID does not exist" Oct 30 17:27:26.679449 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:26.679378 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7"] Oct 30 17:27:26.684017 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:26.683994 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-420ff3-5fd5697cd8-c72b7"] Oct 30 17:27:27.806700 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:27.806650 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb294877-b8b7-45a5-b943-fc62473cedd7" path="/var/lib/kubelet/pods/cb294877-b8b7-45a5-b943-fc62473cedd7/volumes" Oct 30 17:27:30.436626 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:30.436570 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" podUID="1c826c68-020f-4496-8c43-a78ba51f4e9c" containerName="payload-processing" containerID="cri-o://d97538a4feb421b23110b462a2e044000789b0a4d622a6e86619c930da2b2ed9" gracePeriod=25 Oct 30 17:27:31.035209 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.035171 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d"] Oct 30 17:27:31.070261 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.070232 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf"] Oct 30 17:27:31.375788 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.375684 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf"] Oct 30 17:27:31.375965 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.375927 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" podUID="2087171b-22dc-444f-9f62-dd3003edbff9" containerName="maas-api" containerID="cri-o://eb21274d861f8e00cde5d61b85b14db89a69d68848fe7392356af7431caf62f7" gracePeriod=30 Oct 30 17:27:31.632563 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.632499 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 17:27:31.678375 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.678342 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2087171b-22dc-444f-9f62-dd3003edbff9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2087171b-22dc-444f-9f62-dd3003edbff9-maas-api-tls\") pod \"2087171b-22dc-444f-9f62-dd3003edbff9\" (UID: \"2087171b-22dc-444f-9f62-dd3003edbff9\") " Oct 30 17:27:31.678599 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.678439 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2087171b-22dc-444f-9f62-dd3003edbff9-kube-api-access-j7kvt\" (UniqueName: \"kubernetes.io/projected/2087171b-22dc-444f-9f62-dd3003edbff9-kube-api-access-j7kvt\") pod \"2087171b-22dc-444f-9f62-dd3003edbff9\" (UID: \"2087171b-22dc-444f-9f62-dd3003edbff9\") " Oct 30 17:27:31.678599 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.678528 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2087171b-22dc-444f-9f62-dd3003edbff9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2087171b-22dc-444f-9f62-dd3003edbff9-metrics-certs\") pod \"2087171b-22dc-444f-9f62-dd3003edbff9\" (UID: \"2087171b-22dc-444f-9f62-dd3003edbff9\") " Oct 30 17:27:31.678599 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.678578 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2087171b-22dc-444f-9f62-dd3003edbff9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2087171b-22dc-444f-9f62-dd3003edbff9-openshift-service-ca\") pod \"2087171b-22dc-444f-9f62-dd3003edbff9\" (UID: \"2087171b-22dc-444f-9f62-dd3003edbff9\") " Oct 30 17:27:31.680599 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.680543 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2087171b-22dc-444f-9f62-dd3003edbff9-openshift-service-ca" pod "2087171b-22dc-444f-9f62-dd3003edbff9" (UID: "2087171b-22dc-444f-9f62-dd3003edbff9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:31.682397 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.682361 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2087171b-22dc-444f-9f62-dd3003edbff9-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:31.683246 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.683220 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2087171b-22dc-444f-9f62-dd3003edbff9-maas-api-tls" pod "2087171b-22dc-444f-9f62-dd3003edbff9" (UID: "2087171b-22dc-444f-9f62-dd3003edbff9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:31.683369 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.683298 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2087171b-22dc-444f-9f62-dd3003edbff9-metrics-certs" pod "2087171b-22dc-444f-9f62-dd3003edbff9" (UID: "2087171b-22dc-444f-9f62-dd3003edbff9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:31.683670 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.683649 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2087171b-22dc-444f-9f62-dd3003edbff9-kube-api-access-j7kvt" pod "2087171b-22dc-444f-9f62-dd3003edbff9" (UID: "2087171b-22dc-444f-9f62-dd3003edbff9"). InnerVolumeSpecName "kube-api-access-j7kvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:31.691135 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.688461 4861 generic.go:356] "Generic (PLEG): container finished" podID="2087171b-22dc-444f-9f62-dd3003edbff9" containerID="eb21274d861f8e00cde5d61b85b14db89a69d68848fe7392356af7431caf62f7" exitCode=0 Oct 30 17:27:31.691135 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.688555 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" event={"ID":"2087171b-22dc-444f-9f62-dd3003edbff9","Type":"ContainerDied","Data":"eb21274d861f8e00cde5d61b85b14db89a69d68848fe7392356af7431caf62f7"} Oct 30 17:27:31.691135 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.688598 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" event={"ID":"2087171b-22dc-444f-9f62-dd3003edbff9","Type":"ContainerDied","Data":"7552f3fc910846d3e1b55f33d42b71376f3abff72bff84e9797ff7bc2c353d46"} Oct 30 17:27:31.691135 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.688622 4861 scope.go:122] "RemoveContainer" containerID="eb21274d861f8e00cde5d61b85b14db89a69d68848fe7392356af7431caf62f7" Oct 30 17:27:31.691135 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.688771 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf" Oct 30 17:27:31.730204 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.730179 4861 scope.go:122] "RemoveContainer" containerID="eb21274d861f8e00cde5d61b85b14db89a69d68848fe7392356af7431caf62f7" Oct 30 17:27:31.730601 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:31.730577 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb21274d861f8e00cde5d61b85b14db89a69d68848fe7392356af7431caf62f7\": container with ID starting with eb21274d861f8e00cde5d61b85b14db89a69d68848fe7392356af7431caf62f7 not found: ID does not exist" containerID="eb21274d861f8e00cde5d61b85b14db89a69d68848fe7392356af7431caf62f7" Oct 30 17:27:31.730667 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.730609 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb21274d861f8e00cde5d61b85b14db89a69d68848fe7392356af7431caf62f7"} err="failed to get container status \"eb21274d861f8e00cde5d61b85b14db89a69d68848fe7392356af7431caf62f7\": rpc error: code = NotFound desc = could not find container \"eb21274d861f8e00cde5d61b85b14db89a69d68848fe7392356af7431caf62f7\": container with ID starting with eb21274d861f8e00cde5d61b85b14db89a69d68848fe7392356af7431caf62f7 not found: ID does not exist" Oct 30 17:27:31.752563 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.752532 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf"] Oct 30 17:27:31.756183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.756160 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-06e7f6-7d7dfb4dcc-dggbf"] Oct 30 17:27:31.783427 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.783394 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2087171b-22dc-444f-9f62-dd3003edbff9-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:31.783427 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.783418 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2087171b-22dc-444f-9f62-dd3003edbff9-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:31.783427 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.783427 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7kvt\" (UniqueName: \"kubernetes.io/projected/2087171b-22dc-444f-9f62-dd3003edbff9-kube-api-access-j7kvt\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:31.812558 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:31.812237 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2087171b-22dc-444f-9f62-dd3003edbff9" path="/var/lib/kubelet/pods/2087171b-22dc-444f-9f62-dd3003edbff9/volumes" Oct 30 17:27:34.724702 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:34.724644 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" containerID="cri-o://73176821c7b674ca80e171e87d58797edb1a6486e06ba3b83755ccb5b3f0b686" gracePeriod=45 Oct 30 17:27:34.988669 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:34.988610 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 17:27:35.020323 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.020289 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-tls-certs\" (UniqueName: \"kubernetes.io/secret/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-tls-certs\") pod \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " Oct 30 17:27:35.020501 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.020340 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-kube-api-access-tg9qt\" (UniqueName: \"kubernetes.io/projected/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-kube-api-access-tg9qt\") pod \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " Oct 30 17:27:35.020575 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.020515 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-dshm\") pod \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " Oct 30 17:27:35.020575 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.020559 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-model-cache\") pod \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " Oct 30 17:27:35.020738 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.020661 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-home\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-home\") pod \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " Oct 30 17:27:35.020814 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.020715 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-tmp-dir\") pod \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\" (UID: \"f66112b3-864b-46bf-a8dd-3f22c04f8d3f\") " Oct 30 17:27:35.021008 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.020975 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-model-cache" pod "f66112b3-864b-46bf-a8dd-3f22c04f8d3f" (UID: "f66112b3-864b-46bf-a8dd-3f22c04f8d3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:35.021220 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.021199 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-tmp-dir" pod "f66112b3-864b-46bf-a8dd-3f22c04f8d3f" (UID: "f66112b3-864b-46bf-a8dd-3f22c04f8d3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:35.021320 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.021265 4861 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.021384 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.021368 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-home" pod "f66112b3-864b-46bf-a8dd-3f22c04f8d3f" (UID: "f66112b3-864b-46bf-a8dd-3f22c04f8d3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:35.030772 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.023710 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-kube-api-access-tg9qt" pod "f66112b3-864b-46bf-a8dd-3f22c04f8d3f" (UID: "f66112b3-864b-46bf-a8dd-3f22c04f8d3f"). InnerVolumeSpecName "kube-api-access-tg9qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:35.030772 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.024092 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-tls-certs" pod "f66112b3-864b-46bf-a8dd-3f22c04f8d3f" (UID: "f66112b3-864b-46bf-a8dd-3f22c04f8d3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:35.030772 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.026087 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-dshm" pod "f66112b3-864b-46bf-a8dd-3f22c04f8d3f" (UID: "f66112b3-864b-46bf-a8dd-3f22c04f8d3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:35.122694 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.122662 4861 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.122694 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.122691 4861 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.122694 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.122700 4861 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.122949 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.122710 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg9qt\" (UniqueName: \"kubernetes.io/projected/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-kube-api-access-tg9qt\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.122949 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.122735 4861 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66112b3-864b-46bf-a8dd-3f22c04f8d3f-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:35.715353 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.715316 4861 generic.go:356] "Generic (PLEG): container finished" podID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerID="73176821c7b674ca80e171e87d58797edb1a6486e06ba3b83755ccb5b3f0b686" exitCode=0 Oct 30 17:27:35.715532 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.715375 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" Oct 30 17:27:35.715532 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.715424 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" event={"ID":"f66112b3-864b-46bf-a8dd-3f22c04f8d3f","Type":"ContainerDied","Data":"73176821c7b674ca80e171e87d58797edb1a6486e06ba3b83755ccb5b3f0b686"} Oct 30 17:27:35.715532 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.715459 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn" event={"ID":"f66112b3-864b-46bf-a8dd-3f22c04f8d3f","Type":"ContainerDied","Data":"72ec8bf250709309f3320badba59b780edd04aa94c82e25de058839e702ea6b2"} Oct 30 17:27:35.715532 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.715480 4861 scope.go:122] "RemoveContainer" containerID="73176821c7b674ca80e171e87d58797edb1a6486e06ba3b83755ccb5b3f0b686" Oct 30 17:27:35.726777 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.726532 4861 scope.go:122] "RemoveContainer" containerID="73176821c7b674ca80e171e87d58797edb1a6486e06ba3b83755ccb5b3f0b686" Oct 30 17:27:35.727034 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:35.726834 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73176821c7b674ca80e171e87d58797edb1a6486e06ba3b83755ccb5b3f0b686\": container with ID starting with 73176821c7b674ca80e171e87d58797edb1a6486e06ba3b83755ccb5b3f0b686 not found: ID does not exist" containerID="73176821c7b674ca80e171e87d58797edb1a6486e06ba3b83755ccb5b3f0b686" Oct 30 17:27:35.727034 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.726860 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73176821c7b674ca80e171e87d58797edb1a6486e06ba3b83755ccb5b3f0b686"} err="failed to get container status \"73176821c7b674ca80e171e87d58797edb1a6486e06ba3b83755ccb5b3f0b686\": rpc error: code = NotFound desc = could not find container \"73176821c7b674ca80e171e87d58797edb1a6486e06ba3b83755ccb5b3f0b686\": container with ID starting with 73176821c7b674ca80e171e87d58797edb1a6486e06ba3b83755ccb5b3f0b686 not found: ID does not exist" Oct 30 17:27:35.773552 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.773529 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn"] Oct 30 17:27:35.787681 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.787659 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-18519c/premium-w2-18519c-kserve-58bf5cb4d5-fkgvn"] Oct 30 17:27:35.806910 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:35.806885 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" path="/var/lib/kubelet/pods/f66112b3-864b-46bf-a8dd-3f22c04f8d3f/volumes" Oct 30 17:27:36.035586 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.035547 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" podUID="71e9a70b-d593-49ba-82d4-0a022e09197a" containerName="payload-processing" containerID="cri-o://3773b154f04e6864172825bc04faad2051a7d8baae036b6e078f25c8340bccec" gracePeriod=25 Oct 30 17:27:36.071148 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.071109 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" podUID="932e0df6-6b3d-4904-a353-ae7faf5d2544" containerName="payload-pre-processing" containerID="cri-o://6fde81dc89bdfb504e5f0747ae3224d8595625219f70f5ebfe6f68ee23b7dc7f" gracePeriod=25 Oct 30 17:27:36.343643 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.343608 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" Oct 30 17:27:36.377041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.377017 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" Oct 30 17:27:36.437288 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.437256 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e9a70b-d593-49ba-82d4-0a022e09197a-kube-api-access-bz8g4\" (UniqueName: \"kubernetes.io/projected/71e9a70b-d593-49ba-82d4-0a022e09197a-kube-api-access-bz8g4\") pod \"71e9a70b-d593-49ba-82d4-0a022e09197a\" (UID: \"71e9a70b-d593-49ba-82d4-0a022e09197a\") " Oct 30 17:27:36.437473 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.437366 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932e0df6-6b3d-4904-a353-ae7faf5d2544-kube-api-access-mp2jl\" (UniqueName: \"kubernetes.io/projected/932e0df6-6b3d-4904-a353-ae7faf5d2544-kube-api-access-mp2jl\") pod \"932e0df6-6b3d-4904-a353-ae7faf5d2544\" (UID: \"932e0df6-6b3d-4904-a353-ae7faf5d2544\") " Oct 30 17:27:36.437473 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.437393 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932e0df6-6b3d-4904-a353-ae7faf5d2544-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/932e0df6-6b3d-4904-a353-ae7faf5d2544-plugins-config-volume\") pod \"932e0df6-6b3d-4904-a353-ae7faf5d2544\" (UID: \"932e0df6-6b3d-4904-a353-ae7faf5d2544\") " Oct 30 17:27:36.437473 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.437443 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71e9a70b-d593-49ba-82d4-0a022e09197a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e9a70b-d593-49ba-82d4-0a022e09197a-plugins-config-volume\") pod \"71e9a70b-d593-49ba-82d4-0a022e09197a\" (UID: \"71e9a70b-d593-49ba-82d4-0a022e09197a\") " Oct 30 17:27:36.437855 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.437831 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932e0df6-6b3d-4904-a353-ae7faf5d2544-plugins-config-volume" pod "932e0df6-6b3d-4904-a353-ae7faf5d2544" (UID: "932e0df6-6b3d-4904-a353-ae7faf5d2544"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:36.437959 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.437868 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e9a70b-d593-49ba-82d4-0a022e09197a-plugins-config-volume" pod "71e9a70b-d593-49ba-82d4-0a022e09197a" (UID: "71e9a70b-d593-49ba-82d4-0a022e09197a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:36.440067 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.440038 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e9a70b-d593-49ba-82d4-0a022e09197a-kube-api-access-bz8g4" pod "71e9a70b-d593-49ba-82d4-0a022e09197a" (UID: "71e9a70b-d593-49ba-82d4-0a022e09197a"). InnerVolumeSpecName "kube-api-access-bz8g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:36.449763 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.447336 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932e0df6-6b3d-4904-a353-ae7faf5d2544-kube-api-access-mp2jl" pod "932e0df6-6b3d-4904-a353-ae7faf5d2544" (UID: "932e0df6-6b3d-4904-a353-ae7faf5d2544"). InnerVolumeSpecName "kube-api-access-mp2jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:36.539674 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.539631 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz8g4\" (UniqueName: \"kubernetes.io/projected/71e9a70b-d593-49ba-82d4-0a022e09197a-kube-api-access-bz8g4\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:36.539674 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.539679 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp2jl\" (UniqueName: \"kubernetes.io/projected/932e0df6-6b3d-4904-a353-ae7faf5d2544-kube-api-access-mp2jl\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:36.539943 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.539694 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/932e0df6-6b3d-4904-a353-ae7faf5d2544-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:36.539943 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.539710 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71e9a70b-d593-49ba-82d4-0a022e09197a-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:36.731368 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.731306 4861 generic.go:356] "Generic (PLEG): container finished" podID="932e0df6-6b3d-4904-a353-ae7faf5d2544" containerID="6fde81dc89bdfb504e5f0747ae3224d8595625219f70f5ebfe6f68ee23b7dc7f" exitCode=0 Oct 30 17:27:36.731874 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.731389 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" event={"ID":"932e0df6-6b3d-4904-a353-ae7faf5d2544","Type":"ContainerDied","Data":"6fde81dc89bdfb504e5f0747ae3224d8595625219f70f5ebfe6f68ee23b7dc7f"} Oct 30 17:27:36.731874 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.731454 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" event={"ID":"932e0df6-6b3d-4904-a353-ae7faf5d2544","Type":"ContainerDied","Data":"8c3c3844b0d4123357d10e51c11946a690d723d3fc7d3bf9c4f3e4dc88b15564"} Oct 30 17:27:36.731874 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.731488 4861 scope.go:122] "RemoveContainer" containerID="6fde81dc89bdfb504e5f0747ae3224d8595625219f70f5ebfe6f68ee23b7dc7f" Oct 30 17:27:36.731874 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.731624 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf" Oct 30 17:27:36.745389 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.745348 4861 generic.go:356] "Generic (PLEG): container finished" podID="71e9a70b-d593-49ba-82d4-0a022e09197a" containerID="3773b154f04e6864172825bc04faad2051a7d8baae036b6e078f25c8340bccec" exitCode=0 Oct 30 17:27:36.745533 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.745447 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" event={"ID":"71e9a70b-d593-49ba-82d4-0a022e09197a","Type":"ContainerDied","Data":"3773b154f04e6864172825bc04faad2051a7d8baae036b6e078f25c8340bccec"} Oct 30 17:27:36.745533 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.745501 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" event={"ID":"71e9a70b-d593-49ba-82d4-0a022e09197a","Type":"ContainerDied","Data":"bc5401c10a6b0d6af4226024d876bc8f76c97ae2c098f2c20cd59c73c0838127"} Oct 30 17:27:36.745680 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.745576 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d" Oct 30 17:27:36.748928 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.748763 4861 scope.go:122] "RemoveContainer" containerID="6fde81dc89bdfb504e5f0747ae3224d8595625219f70f5ebfe6f68ee23b7dc7f" Oct 30 17:27:36.749101 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:36.749074 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fde81dc89bdfb504e5f0747ae3224d8595625219f70f5ebfe6f68ee23b7dc7f\": container with ID starting with 6fde81dc89bdfb504e5f0747ae3224d8595625219f70f5ebfe6f68ee23b7dc7f not found: ID does not exist" containerID="6fde81dc89bdfb504e5f0747ae3224d8595625219f70f5ebfe6f68ee23b7dc7f" Oct 30 17:27:36.749188 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.749112 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fde81dc89bdfb504e5f0747ae3224d8595625219f70f5ebfe6f68ee23b7dc7f"} err="failed to get container status \"6fde81dc89bdfb504e5f0747ae3224d8595625219f70f5ebfe6f68ee23b7dc7f\": rpc error: code = NotFound desc = could not find container \"6fde81dc89bdfb504e5f0747ae3224d8595625219f70f5ebfe6f68ee23b7dc7f\": container with ID starting with 6fde81dc89bdfb504e5f0747ae3224d8595625219f70f5ebfe6f68ee23b7dc7f not found: ID does not exist" Oct 30 17:27:36.749188 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.749134 4861 scope.go:122] "RemoveContainer" containerID="3773b154f04e6864172825bc04faad2051a7d8baae036b6e078f25c8340bccec" Oct 30 17:27:36.761055 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.761028 4861 scope.go:122] "RemoveContainer" containerID="3773b154f04e6864172825bc04faad2051a7d8baae036b6e078f25c8340bccec" Oct 30 17:27:36.761367 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:36.761346 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3773b154f04e6864172825bc04faad2051a7d8baae036b6e078f25c8340bccec\": container with ID starting with 3773b154f04e6864172825bc04faad2051a7d8baae036b6e078f25c8340bccec not found: ID does not exist" containerID="3773b154f04e6864172825bc04faad2051a7d8baae036b6e078f25c8340bccec" Oct 30 17:27:36.761431 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.761378 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3773b154f04e6864172825bc04faad2051a7d8baae036b6e078f25c8340bccec"} err="failed to get container status \"3773b154f04e6864172825bc04faad2051a7d8baae036b6e078f25c8340bccec\": rpc error: code = NotFound desc = could not find container \"3773b154f04e6864172825bc04faad2051a7d8baae036b6e078f25c8340bccec\": container with ID starting with 3773b154f04e6864172825bc04faad2051a7d8baae036b6e078f25c8340bccec not found: ID does not exist" Oct 30 17:27:36.809401 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.809365 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf"] Oct 30 17:27:36.814110 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.814077 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-06e7f6-65754ff4bc-px4jf"] Oct 30 17:27:36.864995 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.864963 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d"] Oct 30 17:27:36.874164 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:36.874127 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-06e7f6-68c5cdc86b-fg46d"] Oct 30 17:27:37.807733 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:37.807684 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e9a70b-d593-49ba-82d4-0a022e09197a" path="/var/lib/kubelet/pods/71e9a70b-d593-49ba-82d4-0a022e09197a/volumes" Oct 30 17:27:37.808128 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:37.808100 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="932e0df6-6b3d-4904-a353-ae7faf5d2544" path="/var/lib/kubelet/pods/932e0df6-6b3d-4904-a353-ae7faf5d2544/volumes" Oct 30 17:27:37.876766 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:37.876715 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" podUID="1c826c68-020f-4496-8c43-a78ba51f4e9c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:27:40.558543 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:40.558502 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89"] Oct 30 17:27:40.559051 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:40.558773 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" podUID="40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" containerName="istio-proxy" containerID="cri-o://a542a3e78a6d628f01397c66eacde0362422c9bbc2b0b2ab6a84d563204696c2" gracePeriod=30 Oct 30 17:27:45.805897 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.805866 4861 generic.go:356] "Generic (PLEG): container finished" podID="40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" containerID="a542a3e78a6d628f01397c66eacde0362422c9bbc2b0b2ab6a84d563204696c2" exitCode=0 Oct 30 17:27:45.806885 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.806855 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" event={"ID":"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf","Type":"ContainerDied","Data":"a542a3e78a6d628f01397c66eacde0362422c9bbc2b0b2ab6a84d563204696c2"} Oct 30 17:27:45.807061 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.806900 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" event={"ID":"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf","Type":"ContainerDied","Data":"07a6c7680612f87ef21042e687bbe8011d744ff5d2760d38ff1de6aed12ebd1d"} Oct 30 17:27:45.807061 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.806916 4861 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07a6c7680612f87ef21042e687bbe8011d744ff5d2760d38ff1de6aed12ebd1d" Oct 30 17:27:45.818791 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.818769 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:27:45.937821 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.937741 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq"] Oct 30 17:27:45.942835 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.942808 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth"] Oct 30 17:27:45.950634 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.950610 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-kube-api-access-48fd7\" (UniqueName: \"kubernetes.io/projected/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-kube-api-access-48fd7\") pod \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " Oct 30 17:27:45.950796 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.950676 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-workload-certs\") pod \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " Oct 30 17:27:45.950796 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.950772 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-podinfo\") pod \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " Oct 30 17:27:45.950931 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.950810 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-workload-socket\") pod \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " Oct 30 17:27:45.950931 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.950882 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-data\") pod \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " Oct 30 17:27:45.951058 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.950929 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-credential-socket\") pod \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " Oct 30 17:27:45.951058 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.950978 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-token\" (UniqueName: \"kubernetes.io/projected/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-token\") pod \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " Oct 30 17:27:45.951058 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.951044 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istiod-ca-cert\") pod \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " Oct 30 17:27:45.951217 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.951076 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-envoy\") pod \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\" (UID: \"40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf\") " Oct 30 17:27:45.951403 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.951062 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-workload-certs" pod "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" (UID: "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:45.951547 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.951156 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-data" pod "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" (UID: "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:45.951547 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.951146 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-workload-socket" pod "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" (UID: "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:45.951547 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.951236 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-credential-socket" pod "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" (UID: "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:45.951900 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.951761 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istiod-ca-cert" pod "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" (UID: "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:45.952547 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.952525 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:45.952635 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.952551 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:45.952635 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.952566 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:45.952635 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.952580 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:45.952635 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.952594 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:45.954087 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.954052 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-podinfo" pod "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" (UID: "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:27:45.954746 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.954688 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-kube-api-access-48fd7" pod "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" (UID: "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf"). InnerVolumeSpecName "kube-api-access-48fd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:45.954860 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.954808 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-token" pod "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" (UID: "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:45.954985 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:45.954969 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-envoy" pod "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" (UID: "40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:27:46.053481 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:46.053452 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:46.053481 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:46.053477 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48fd7\" (UniqueName: \"kubernetes.io/projected/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-kube-api-access-48fd7\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:46.053481 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:46.053486 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:46.053678 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:46.053494 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:46.812440 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:46.811858 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89" Oct 30 17:27:46.885028 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:46.884995 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89"] Oct 30 17:27:46.887308 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:46.887283 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-06e7f6-openshift-default-6fb844f87c-hdq89"] Oct 30 17:27:47.816494 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:47.816452 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" path="/var/lib/kubelet/pods/40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf/volumes" Oct 30 17:27:47.876135 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:47.876107 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" podUID="1c826c68-020f-4496-8c43-a78ba51f4e9c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:27:50.291809 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:50.291774 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:27:50.516669 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.516647 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 17:27:50.553537 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.553475 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 17:27:50.706217 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.706190 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" Oct 30 17:27:50.802451 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.802422 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c826c68-020f-4496-8c43-a78ba51f4e9c-kube-api-access-cmtgg\" (UniqueName: \"kubernetes.io/projected/1c826c68-020f-4496-8c43-a78ba51f4e9c-kube-api-access-cmtgg\") pod \"1c826c68-020f-4496-8c43-a78ba51f4e9c\" (UID: \"1c826c68-020f-4496-8c43-a78ba51f4e9c\") " Oct 30 17:27:50.802683 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.802533 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c826c68-020f-4496-8c43-a78ba51f4e9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c826c68-020f-4496-8c43-a78ba51f4e9c-plugins-config-volume\") pod \"1c826c68-020f-4496-8c43-a78ba51f4e9c\" (UID: \"1c826c68-020f-4496-8c43-a78ba51f4e9c\") " Oct 30 17:27:50.802987 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.802954 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c826c68-020f-4496-8c43-a78ba51f4e9c-plugins-config-volume" pod "1c826c68-020f-4496-8c43-a78ba51f4e9c" (UID: "1c826c68-020f-4496-8c43-a78ba51f4e9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:50.805543 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.805480 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c826c68-020f-4496-8c43-a78ba51f4e9c-kube-api-access-cmtgg" pod "1c826c68-020f-4496-8c43-a78ba51f4e9c" (UID: "1c826c68-020f-4496-8c43-a78ba51f4e9c"). InnerVolumeSpecName "kube-api-access-cmtgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:50.842822 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.842791 4861 generic.go:356] "Generic (PLEG): container finished" podID="1c826c68-020f-4496-8c43-a78ba51f4e9c" containerID="d97538a4feb421b23110b462a2e044000789b0a4d622a6e86619c930da2b2ed9" exitCode=0 Oct 30 17:27:50.842948 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.842839 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" Oct 30 17:27:50.842948 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.842855 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" event={"ID":"1c826c68-020f-4496-8c43-a78ba51f4e9c","Type":"ContainerDied","Data":"d97538a4feb421b23110b462a2e044000789b0a4d622a6e86619c930da2b2ed9"} Oct 30 17:27:50.842948 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.842913 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78" event={"ID":"1c826c68-020f-4496-8c43-a78ba51f4e9c","Type":"ContainerDied","Data":"f776eb41693f60433146169aa18291e9a318836a16002e2ca9ef7dcaff058f83"} Oct 30 17:27:50.842948 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.842942 4861 scope.go:122] "RemoveContainer" containerID="d97538a4feb421b23110b462a2e044000789b0a4d622a6e86619c930da2b2ed9" Oct 30 17:27:50.853874 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.853848 4861 scope.go:122] "RemoveContainer" containerID="d97538a4feb421b23110b462a2e044000789b0a4d622a6e86619c930da2b2ed9" Oct 30 17:27:50.854212 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:50.854188 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97538a4feb421b23110b462a2e044000789b0a4d622a6e86619c930da2b2ed9\": container with ID starting with d97538a4feb421b23110b462a2e044000789b0a4d622a6e86619c930da2b2ed9 not found: ID does not exist" containerID="d97538a4feb421b23110b462a2e044000789b0a4d622a6e86619c930da2b2ed9" Oct 30 17:27:50.854275 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.854223 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97538a4feb421b23110b462a2e044000789b0a4d622a6e86619c930da2b2ed9"} err="failed to get container status \"d97538a4feb421b23110b462a2e044000789b0a4d622a6e86619c930da2b2ed9\": rpc error: code = NotFound desc = could not find container \"d97538a4feb421b23110b462a2e044000789b0a4d622a6e86619c930da2b2ed9\": container with ID starting with d97538a4feb421b23110b462a2e044000789b0a4d622a6e86619c930da2b2ed9 not found: ID does not exist" Oct 30 17:27:50.899156 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.899131 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78"] Oct 30 17:27:50.902680 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.902649 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-18519c-6b9cfdb59b-9vz78"] Oct 30 17:27:50.904332 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.904300 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmtgg\" (UniqueName: \"kubernetes.io/projected/1c826c68-020f-4496-8c43-a78ba51f4e9c-kube-api-access-cmtgg\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:50.904412 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:50.904331 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c826c68-020f-4496-8c43-a78ba51f4e9c-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:51.807266 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:51.807232 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c826c68-020f-4496-8c43-a78ba51f4e9c" path="/var/lib/kubelet/pods/1c826c68-020f-4496-8c43-a78ba51f4e9c/volumes" Oct 30 17:27:53.019508 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:53.019471 4861 scope.go:122] "RemoveContainer" containerID="a542a3e78a6d628f01397c66eacde0362422c9bbc2b0b2ab6a84d563204696c2" Oct 30 17:27:53.678243 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:53.678210 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq"] Oct 30 17:27:53.744646 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:53.744609 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28"] Oct 30 17:27:58.286127 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.286074 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f"] Oct 30 17:27:58.286638 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.286417 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" podUID="f988c028-5967-4436-a8cd-30af1e920b40" containerName="maas-api" containerID="cri-o://a0b5fa829a0d52f40f28a9ea8b8802943a54208491b8d8aed3e6012a9a239b47" gracePeriod=30 Oct 30 17:27:58.555406 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.555108 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:27:58.679140 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.679105 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" podUID="0f099e88-ce47-485b-a7f6-2726c8f128c1" containerName="payload-processing" containerID="cri-o://5869c05076ce940727db1ad05baacb2608e8c662abb7816d5d6cb4a1d9c1b9f3" gracePeriod=25 Oct 30 17:27:58.699246 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.699216 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f988c028-5967-4436-a8cd-30af1e920b40-kube-api-access-l5872\" (UniqueName: \"kubernetes.io/projected/f988c028-5967-4436-a8cd-30af1e920b40-kube-api-access-l5872\") pod \"f988c028-5967-4436-a8cd-30af1e920b40\" (UID: \"f988c028-5967-4436-a8cd-30af1e920b40\") " Oct 30 17:27:58.699408 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.699295 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f988c028-5967-4436-a8cd-30af1e920b40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f988c028-5967-4436-a8cd-30af1e920b40-maas-api-tls\") pod \"f988c028-5967-4436-a8cd-30af1e920b40\" (UID: \"f988c028-5967-4436-a8cd-30af1e920b40\") " Oct 30 17:27:58.699482 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.699452 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f988c028-5967-4436-a8cd-30af1e920b40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f988c028-5967-4436-a8cd-30af1e920b40-openshift-service-ca\") pod \"f988c028-5967-4436-a8cd-30af1e920b40\" (UID: \"f988c028-5967-4436-a8cd-30af1e920b40\") " Oct 30 17:27:58.699578 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.699560 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f988c028-5967-4436-a8cd-30af1e920b40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f988c028-5967-4436-a8cd-30af1e920b40-metrics-certs\") pod \"f988c028-5967-4436-a8cd-30af1e920b40\" (UID: \"f988c028-5967-4436-a8cd-30af1e920b40\") " Oct 30 17:27:58.700231 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.699973 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f988c028-5967-4436-a8cd-30af1e920b40-openshift-service-ca" pod "f988c028-5967-4436-a8cd-30af1e920b40" (UID: "f988c028-5967-4436-a8cd-30af1e920b40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:27:58.700231 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.700197 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f988c028-5967-4436-a8cd-30af1e920b40-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:58.701896 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.701865 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f988c028-5967-4436-a8cd-30af1e920b40-kube-api-access-l5872" pod "f988c028-5967-4436-a8cd-30af1e920b40" (UID: "f988c028-5967-4436-a8cd-30af1e920b40"). InnerVolumeSpecName "kube-api-access-l5872". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:27:58.702132 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.702106 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f988c028-5967-4436-a8cd-30af1e920b40-metrics-certs" pod "f988c028-5967-4436-a8cd-30af1e920b40" (UID: "f988c028-5967-4436-a8cd-30af1e920b40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:58.702281 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.702131 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f988c028-5967-4436-a8cd-30af1e920b40-maas-api-tls" pod "f988c028-5967-4436-a8cd-30af1e920b40" (UID: "f988c028-5967-4436-a8cd-30af1e920b40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:27:58.745131 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.745085 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" podUID="b9c6277c-c197-4b2a-945b-9c2553114762" containerName="payload-pre-processing" containerID="cri-o://5b29d396eb2dd053e6efc4044eeef4053bef212b1ae834ae26eb3e82d77efb40" gracePeriod=25 Oct 30 17:27:58.801600 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.801560 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f988c028-5967-4436-a8cd-30af1e920b40-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:58.801600 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.801593 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f988c028-5967-4436-a8cd-30af1e920b40-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:58.801600 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.801603 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5872\" (UniqueName: \"kubernetes.io/projected/f988c028-5967-4436-a8cd-30af1e920b40-kube-api-access-l5872\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:27:58.901935 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.901858 4861 generic.go:356] "Generic (PLEG): container finished" podID="f988c028-5967-4436-a8cd-30af1e920b40" containerID="a0b5fa829a0d52f40f28a9ea8b8802943a54208491b8d8aed3e6012a9a239b47" exitCode=0 Oct 30 17:27:58.901935 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.901910 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" Oct 30 17:27:58.902158 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.901909 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" event={"ID":"f988c028-5967-4436-a8cd-30af1e920b40","Type":"ContainerDied","Data":"a0b5fa829a0d52f40f28a9ea8b8802943a54208491b8d8aed3e6012a9a239b47"} Oct 30 17:27:58.902158 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.902040 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f" event={"ID":"f988c028-5967-4436-a8cd-30af1e920b40","Type":"ContainerDied","Data":"7cbab083596685678fd9b54c7c20aa8e3760c71b9eb1006cebb0d7a9c61ca02c"} Oct 30 17:27:58.902158 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.902066 4861 scope.go:122] "RemoveContainer" containerID="a0b5fa829a0d52f40f28a9ea8b8802943a54208491b8d8aed3e6012a9a239b47" Oct 30 17:27:58.915684 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.915630 4861 scope.go:122] "RemoveContainer" containerID="a0b5fa829a0d52f40f28a9ea8b8802943a54208491b8d8aed3e6012a9a239b47" Oct 30 17:27:58.915951 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:27:58.915927 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b5fa829a0d52f40f28a9ea8b8802943a54208491b8d8aed3e6012a9a239b47\": container with ID starting with a0b5fa829a0d52f40f28a9ea8b8802943a54208491b8d8aed3e6012a9a239b47 not found: ID does not exist" containerID="a0b5fa829a0d52f40f28a9ea8b8802943a54208491b8d8aed3e6012a9a239b47" Oct 30 17:27:58.916041 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.915962 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b5fa829a0d52f40f28a9ea8b8802943a54208491b8d8aed3e6012a9a239b47"} err="failed to get container status \"a0b5fa829a0d52f40f28a9ea8b8802943a54208491b8d8aed3e6012a9a239b47\": rpc error: code = NotFound desc = could not find container \"a0b5fa829a0d52f40f28a9ea8b8802943a54208491b8d8aed3e6012a9a239b47\": container with ID starting with a0b5fa829a0d52f40f28a9ea8b8802943a54208491b8d8aed3e6012a9a239b47 not found: ID does not exist" Oct 30 17:27:58.979888 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.979860 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f"] Oct 30 17:27:58.992533 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:58.992508 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f94fb5-796b5ccc96-hk78f"] Oct 30 17:27:59.807013 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:27:59.806976 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f988c028-5967-4436-a8cd-30af1e920b40" path="/var/lib/kubelet/pods/f988c028-5967-4436-a8cd-30af1e920b40/volumes" Oct 30 17:28:00.961466 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:00.961363 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" containerID="cri-o://1f5cb58ea1fbe3687ea2af5157c1a3596dbac64700ef488a5fc312eef4539552" gracePeriod=45 Oct 30 17:28:00.966593 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:00.966098 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" containerID="cri-o://2520f862fc2eca9835531159b47679229aaef76f3394e498691b3be6bfe4a74d" gracePeriod=45 Oct 30 17:28:01.231347 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.231310 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 17:28:01.259766 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.259739 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 17:28:01.324135 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.324107 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-dshm\") pod \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " Oct 30 17:28:01.324317 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.324150 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-home\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-home\") pod \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " Oct 30 17:28:01.324317 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.324200 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-tmp-dir\") pod \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " Oct 30 17:28:01.324317 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.324228 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-model-cache\") pod \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " Oct 30 17:28:01.324515 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.324321 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1837e2a6-812b-4200-bee2-21c23c1a8bb9-kube-api-access-ffhm7\" (UniqueName: \"kubernetes.io/projected/1837e2a6-812b-4200-bee2-21c23c1a8bb9-kube-api-access-ffhm7\") pod \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " Oct 30 17:28:01.324515 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.324355 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-home\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-home\") pod \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " Oct 30 17:28:01.324515 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.324391 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-model-cache\") pod \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " Oct 30 17:28:01.324515 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.324417 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-tmp-dir\") pod \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " Oct 30 17:28:01.324515 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.324482 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-kube-api-access-bnv6c\" (UniqueName: \"kubernetes.io/projected/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-kube-api-access-bnv6c\") pod \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " Oct 30 17:28:01.324515 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.324499 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-tmp-dir" pod "1837e2a6-812b-4200-bee2-21c23c1a8bb9" (UID: "1837e2a6-812b-4200-bee2-21c23c1a8bb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:01.324884 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.324510 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-home" pod "7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" (UID: "7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:01.324884 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.324517 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1837e2a6-812b-4200-bee2-21c23c1a8bb9-tls-certs\" (UniqueName: \"kubernetes.io/secret/1837e2a6-812b-4200-bee2-21c23c1a8bb9-tls-certs\") pod \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\" (UID: \"1837e2a6-812b-4200-bee2-21c23c1a8bb9\") " Oct 30 17:28:01.327787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.325160 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-home" pod "1837e2a6-812b-4200-bee2-21c23c1a8bb9" (UID: "1837e2a6-812b-4200-bee2-21c23c1a8bb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:01.327787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.324637 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-tls-certs\") pod \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " Oct 30 17:28:01.327787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.325381 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-dshm\") pod \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\" (UID: \"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc\") " Oct 30 17:28:01.327787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.325408 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-tmp-dir" pod "7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" (UID: "7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:01.327787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.325860 4861 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.327787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.325880 4861 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.327787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.325896 4861 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-home\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.327787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.325910 4861 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-tmp-dir\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.327787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.327304 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-dshm" pod "1837e2a6-812b-4200-bee2-21c23c1a8bb9" (UID: "1837e2a6-812b-4200-bee2-21c23c1a8bb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:01.327787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.327346 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1837e2a6-812b-4200-bee2-21c23c1a8bb9-kube-api-access-ffhm7" pod "1837e2a6-812b-4200-bee2-21c23c1a8bb9" (UID: "1837e2a6-812b-4200-bee2-21c23c1a8bb9"). InnerVolumeSpecName "kube-api-access-ffhm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:01.327787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.327366 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-model-cache" pod "7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" (UID: "7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:01.327787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.327424 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-model-cache" pod "1837e2a6-812b-4200-bee2-21c23c1a8bb9" (UID: "1837e2a6-812b-4200-bee2-21c23c1a8bb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:01.329044 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.328441 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-dshm" pod "7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" (UID: "7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:01.329116 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.329079 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1837e2a6-812b-4200-bee2-21c23c1a8bb9-tls-certs" pod "1837e2a6-812b-4200-bee2-21c23c1a8bb9" (UID: "1837e2a6-812b-4200-bee2-21c23c1a8bb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:28:01.329593 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.329568 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-kube-api-access-bnv6c" pod "7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" (UID: "7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc"). InnerVolumeSpecName "kube-api-access-bnv6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:01.329888 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.329871 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-tls-certs" pod "7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" (UID: "7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:28:01.427127 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.427092 4861 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.427127 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.427125 4861 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.427328 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.427136 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffhm7\" (UniqueName: \"kubernetes.io/projected/1837e2a6-812b-4200-bee2-21c23c1a8bb9-kube-api-access-ffhm7\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.427328 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.427145 4861 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1837e2a6-812b-4200-bee2-21c23c1a8bb9-model-cache\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.427328 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.427155 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnv6c\" (UniqueName: \"kubernetes.io/projected/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-kube-api-access-bnv6c\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.427328 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.427164 4861 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1837e2a6-812b-4200-bee2-21c23c1a8bb9-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.427328 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.427172 4861 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-tls-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.427328 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.427179 4861 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc-dshm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:01.920949 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.920913 4861 generic.go:356] "Generic (PLEG): container finished" podID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerID="1f5cb58ea1fbe3687ea2af5157c1a3596dbac64700ef488a5fc312eef4539552" exitCode=0 Oct 30 17:28:01.921108 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.920969 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" Oct 30 17:28:01.921108 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.920992 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" event={"ID":"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc","Type":"ContainerDied","Data":"1f5cb58ea1fbe3687ea2af5157c1a3596dbac64700ef488a5fc312eef4539552"} Oct 30 17:28:01.921108 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.921040 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq" event={"ID":"7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc","Type":"ContainerDied","Data":"46beef71e434a95c2f9426de50dcaa28585aeeb88d3e77cf252207160b90bed1"} Oct 30 17:28:01.921108 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.921057 4861 scope.go:122] "RemoveContainer" containerID="1f5cb58ea1fbe3687ea2af5157c1a3596dbac64700ef488a5fc312eef4539552" Oct 30 17:28:01.922458 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.922435 4861 generic.go:356] "Generic (PLEG): container finished" podID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerID="2520f862fc2eca9835531159b47679229aaef76f3394e498691b3be6bfe4a74d" exitCode=0 Oct 30 17:28:01.922608 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.922482 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" Oct 30 17:28:01.922608 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.922536 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" event={"ID":"1837e2a6-812b-4200-bee2-21c23c1a8bb9","Type":"ContainerDied","Data":"2520f862fc2eca9835531159b47679229aaef76f3394e498691b3be6bfe4a74d"} Oct 30 17:28:01.922608 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.922573 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth" event={"ID":"1837e2a6-812b-4200-bee2-21c23c1a8bb9","Type":"ContainerDied","Data":"989b5a9c13699f295829abeb4a8383f1db6ad0d4fdb25b2b72ae6436a2b4c0cf"} Oct 30 17:28:01.930616 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.930599 4861 scope.go:122] "RemoveContainer" containerID="1f5cb58ea1fbe3687ea2af5157c1a3596dbac64700ef488a5fc312eef4539552" Oct 30 17:28:01.930948 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:28:01.930925 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f5cb58ea1fbe3687ea2af5157c1a3596dbac64700ef488a5fc312eef4539552\": container with ID starting with 1f5cb58ea1fbe3687ea2af5157c1a3596dbac64700ef488a5fc312eef4539552 not found: ID does not exist" containerID="1f5cb58ea1fbe3687ea2af5157c1a3596dbac64700ef488a5fc312eef4539552" Oct 30 17:28:01.931046 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.930953 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f5cb58ea1fbe3687ea2af5157c1a3596dbac64700ef488a5fc312eef4539552"} err="failed to get container status \"1f5cb58ea1fbe3687ea2af5157c1a3596dbac64700ef488a5fc312eef4539552\": rpc error: code = NotFound desc = could not find container \"1f5cb58ea1fbe3687ea2af5157c1a3596dbac64700ef488a5fc312eef4539552\": container with ID starting with 1f5cb58ea1fbe3687ea2af5157c1a3596dbac64700ef488a5fc312eef4539552 not found: ID does not exist" Oct 30 17:28:01.931046 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.930971 4861 scope.go:122] "RemoveContainer" containerID="2520f862fc2eca9835531159b47679229aaef76f3394e498691b3be6bfe4a74d" Oct 30 17:28:01.939665 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.939642 4861 scope.go:122] "RemoveContainer" containerID="2520f862fc2eca9835531159b47679229aaef76f3394e498691b3be6bfe4a74d" Oct 30 17:28:01.939971 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:28:01.939951 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2520f862fc2eca9835531159b47679229aaef76f3394e498691b3be6bfe4a74d\": container with ID starting with 2520f862fc2eca9835531159b47679229aaef76f3394e498691b3be6bfe4a74d not found: ID does not exist" containerID="2520f862fc2eca9835531159b47679229aaef76f3394e498691b3be6bfe4a74d" Oct 30 17:28:01.940036 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.939977 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2520f862fc2eca9835531159b47679229aaef76f3394e498691b3be6bfe4a74d"} err="failed to get container status \"2520f862fc2eca9835531159b47679229aaef76f3394e498691b3be6bfe4a74d\": rpc error: code = NotFound desc = could not find container \"2520f862fc2eca9835531159b47679229aaef76f3394e498691b3be6bfe4a74d\": container with ID starting with 2520f862fc2eca9835531159b47679229aaef76f3394e498691b3be6bfe4a74d not found: ID does not exist" Oct 30 17:28:01.980953 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.980933 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth"] Oct 30 17:28:01.994440 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:01.994415 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/distinct-w0-f3503b-kserve-5998b8d978-dwhth"] Oct 30 17:28:02.026618 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:02.026591 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq"] Oct 30 17:28:02.030127 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:02.030106 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f3503b/distinct-2-w0-f3503b-kserve-d8f8b5bbf-rd6xq"] Oct 30 17:28:02.362236 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:02.362193 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" podUID="0f099e88-ce47-485b-a7f6-2726c8f128c1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:28:02.362394 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:02.362193 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" podUID="b9c6277c-c197-4b2a-945b-9c2553114762" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:28:03.806002 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:03.805968 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" path="/var/lib/kubelet/pods/1837e2a6-812b-4200-bee2-21c23c1a8bb9/volumes" Oct 30 17:28:03.806399 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:03.806375 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" path="/var/lib/kubelet/pods/7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc/volumes" Oct 30 17:28:12.362579 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:12.362537 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" podUID="0f099e88-ce47-485b-a7f6-2726c8f128c1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:28:12.365412 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:12.362538 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" podUID="b9c6277c-c197-4b2a-945b-9c2553114762" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:28:17.785621 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:17.785574 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7"] Oct 30 17:28:17.786139 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:17.786009 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" podUID="fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" containerName="istio-proxy" containerID="cri-o://0ed9b3b072fac4b32f428cacde8c13d2eaa86031dbe367cb636569ef104fedcf" gracePeriod=30 Oct 30 17:28:17.960683 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:17.960651 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp"] Oct 30 17:28:17.992883 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:17.992853 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj"] Oct 30 17:28:19.006865 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.006838 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" Oct 30 17:28:19.020405 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.020375 4861 generic.go:356] "Generic (PLEG): container finished" podID="0f099e88-ce47-485b-a7f6-2726c8f128c1" containerID="5869c05076ce940727db1ad05baacb2608e8c662abb7816d5d6cb4a1d9c1b9f3" exitCode=0 Oct 30 17:28:19.020606 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.020470 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" Oct 30 17:28:19.020606 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.020464 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" event={"ID":"0f099e88-ce47-485b-a7f6-2726c8f128c1","Type":"ContainerDied","Data":"5869c05076ce940727db1ad05baacb2608e8c662abb7816d5d6cb4a1d9c1b9f3"} Oct 30 17:28:19.020606 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.020597 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq" event={"ID":"0f099e88-ce47-485b-a7f6-2726c8f128c1","Type":"ContainerDied","Data":"94883dafb3d29e97ee7ae7c97526641adcecdad9d22d922bd7ddcee8dbb94eec"} Oct 30 17:28:19.020796 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.020636 4861 scope.go:122] "RemoveContainer" containerID="5869c05076ce940727db1ad05baacb2608e8c662abb7816d5d6cb4a1d9c1b9f3" Oct 30 17:28:19.022508 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.022484 4861 generic.go:356] "Generic (PLEG): container finished" podID="b9c6277c-c197-4b2a-945b-9c2553114762" containerID="5b29d396eb2dd053e6efc4044eeef4053bef212b1ae834ae26eb3e82d77efb40" exitCode=0 Oct 30 17:28:19.022631 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.022557 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" event={"ID":"b9c6277c-c197-4b2a-945b-9c2553114762","Type":"ContainerDied","Data":"5b29d396eb2dd053e6efc4044eeef4053bef212b1ae834ae26eb3e82d77efb40"} Oct 30 17:28:19.038457 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.038437 4861 scope.go:122] "RemoveContainer" containerID="5869c05076ce940727db1ad05baacb2608e8c662abb7816d5d6cb4a1d9c1b9f3" Oct 30 17:28:19.038783 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:28:19.038716 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5869c05076ce940727db1ad05baacb2608e8c662abb7816d5d6cb4a1d9c1b9f3\": container with ID starting with 5869c05076ce940727db1ad05baacb2608e8c662abb7816d5d6cb4a1d9c1b9f3 not found: ID does not exist" containerID="5869c05076ce940727db1ad05baacb2608e8c662abb7816d5d6cb4a1d9c1b9f3" Oct 30 17:28:19.038878 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.038791 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5869c05076ce940727db1ad05baacb2608e8c662abb7816d5d6cb4a1d9c1b9f3"} err="failed to get container status \"5869c05076ce940727db1ad05baacb2608e8c662abb7816d5d6cb4a1d9c1b9f3\": rpc error: code = NotFound desc = could not find container \"5869c05076ce940727db1ad05baacb2608e8c662abb7816d5d6cb4a1d9c1b9f3\": container with ID starting with 5869c05076ce940727db1ad05baacb2608e8c662abb7816d5d6cb4a1d9c1b9f3 not found: ID does not exist" Oct 30 17:28:19.040182 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.040166 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" Oct 30 17:28:19.112493 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.112457 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c6277c-c197-4b2a-945b-9c2553114762-kube-api-access-pxdxn\" (UniqueName: \"kubernetes.io/projected/b9c6277c-c197-4b2a-945b-9c2553114762-kube-api-access-pxdxn\") pod \"b9c6277c-c197-4b2a-945b-9c2553114762\" (UID: \"b9c6277c-c197-4b2a-945b-9c2553114762\") " Oct 30 17:28:19.112684 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.112511 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f099e88-ce47-485b-a7f6-2726c8f128c1-kube-api-access-5v6z4\" (UniqueName: \"kubernetes.io/projected/0f099e88-ce47-485b-a7f6-2726c8f128c1-kube-api-access-5v6z4\") pod \"0f099e88-ce47-485b-a7f6-2726c8f128c1\" (UID: \"0f099e88-ce47-485b-a7f6-2726c8f128c1\") " Oct 30 17:28:19.112684 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.112555 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f099e88-ce47-485b-a7f6-2726c8f128c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f099e88-ce47-485b-a7f6-2726c8f128c1-plugins-config-volume\") pod \"0f099e88-ce47-485b-a7f6-2726c8f128c1\" (UID: \"0f099e88-ce47-485b-a7f6-2726c8f128c1\") " Oct 30 17:28:19.112684 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.112598 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c6277c-c197-4b2a-945b-9c2553114762-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c6277c-c197-4b2a-945b-9c2553114762-plugins-config-volume\") pod \"b9c6277c-c197-4b2a-945b-9c2553114762\" (UID: \"b9c6277c-c197-4b2a-945b-9c2553114762\") " Oct 30 17:28:19.113243 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.113210 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c6277c-c197-4b2a-945b-9c2553114762-plugins-config-volume" pod "b9c6277c-c197-4b2a-945b-9c2553114762" (UID: "b9c6277c-c197-4b2a-945b-9c2553114762"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:28:19.113584 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.113560 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f099e88-ce47-485b-a7f6-2726c8f128c1-plugins-config-volume" pod "0f099e88-ce47-485b-a7f6-2726c8f128c1" (UID: "0f099e88-ce47-485b-a7f6-2726c8f128c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:28:19.114971 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.114942 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c6277c-c197-4b2a-945b-9c2553114762-kube-api-access-pxdxn" pod "b9c6277c-c197-4b2a-945b-9c2553114762" (UID: "b9c6277c-c197-4b2a-945b-9c2553114762"). InnerVolumeSpecName "kube-api-access-pxdxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:19.115107 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.115070 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f099e88-ce47-485b-a7f6-2726c8f128c1-kube-api-access-5v6z4" pod "0f099e88-ce47-485b-a7f6-2726c8f128c1" (UID: "0f099e88-ce47-485b-a7f6-2726c8f128c1"). InnerVolumeSpecName "kube-api-access-5v6z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:19.214143 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.214060 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxdxn\" (UniqueName: \"kubernetes.io/projected/b9c6277c-c197-4b2a-945b-9c2553114762-kube-api-access-pxdxn\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:19.214143 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.214093 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v6z4\" (UniqueName: \"kubernetes.io/projected/0f099e88-ce47-485b-a7f6-2726c8f128c1-kube-api-access-5v6z4\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:19.214143 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.214107 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f099e88-ce47-485b-a7f6-2726c8f128c1-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:19.214143 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.214119 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9c6277c-c197-4b2a-945b-9c2553114762-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:19.368142 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.368110 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq"] Oct 30 17:28:19.381854 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.381824 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f94fb5-5c669bf9d4-7jpnq"] Oct 30 17:28:19.808954 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:19.808917 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f099e88-ce47-485b-a7f6-2726c8f128c1" path="/var/lib/kubelet/pods/0f099e88-ce47-485b-a7f6-2726c8f128c1/volumes" Oct 30 17:28:20.029239 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:20.029207 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" Oct 30 17:28:20.029742 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:20.029236 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28" event={"ID":"b9c6277c-c197-4b2a-945b-9c2553114762","Type":"ContainerDied","Data":"2e3b7c1b7837eece24556aded72bcb127e2b6ebdf659810ed99d600706832a66"} Oct 30 17:28:20.029742 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:20.029289 4861 scope.go:122] "RemoveContainer" containerID="5b29d396eb2dd053e6efc4044eeef4053bef212b1ae834ae26eb3e82d77efb40" Oct 30 17:28:20.085067 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:20.085040 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28"] Oct 30 17:28:20.099017 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:20.098967 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f94fb5-5b558f7bf9-9jk28"] Oct 30 17:28:20.291660 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:28:20.291575 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:28:21.813231 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:21.813188 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c6277c-c197-4b2a-945b-9c2553114762" path="/var/lib/kubelet/pods/b9c6277c-c197-4b2a-945b-9c2553114762/volumes" Oct 30 17:28:22.961101 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:22.961045 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" podUID="ce164706-e748-479e-a1a6-7977e983bc9e" containerName="payload-processing" containerID="cri-o://81b274cea3578b2b0f44f15cb1034c7ad6fba6244210eb9e19b3701bd7a96f9f" gracePeriod=25 Oct 30 17:28:22.993262 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:22.993225 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" podUID="40c65962-30bc-4ea9-b938-4a4917e875cc" containerName="payload-pre-processing" containerID="cri-o://87eccaefa07e400cd38b7ac2c6833f5c30115424fddc38a412fe17abfb84133f" gracePeriod=25 Oct 30 17:28:23.050038 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.050009 4861 generic.go:356] "Generic (PLEG): container finished" podID="fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" containerID="0ed9b3b072fac4b32f428cacde8c13d2eaa86031dbe367cb636569ef104fedcf" exitCode=0 Oct 30 17:28:23.050171 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.050077 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" event={"ID":"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3","Type":"ContainerDied","Data":"0ed9b3b072fac4b32f428cacde8c13d2eaa86031dbe367cb636569ef104fedcf"} Oct 30 17:28:23.050171 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.050126 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" event={"ID":"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3","Type":"ContainerDied","Data":"fd0fe812b82c7b82e9c7ffd6be17dc223044105e95aabe6bcbd0ddcb3f33fc28"} Oct 30 17:28:23.050171 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.050142 4861 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd0fe812b82c7b82e9c7ffd6be17dc223044105e95aabe6bcbd0ddcb3f33fc28" Oct 30 17:28:23.050171 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.050155 4861 scope.go:122] "RemoveContainer" containerID="74aadff8eee521b2089a67530089b6125daf98ab1faa898acd5a866e79aa0e73" Oct 30 17:28:23.058988 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.058932 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 17:28:23.154908 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.154875 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-credential-socket\") pod \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " Oct 30 17:28:23.155092 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.154963 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-podinfo\") pod \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " Oct 30 17:28:23.155092 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.155011 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-envoy\") pod \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " Oct 30 17:28:23.155092 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.155050 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-workload-socket\") pod \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " Oct 30 17:28:23.155092 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.155072 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istiod-ca-cert\") pod \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " Oct 30 17:28:23.155273 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.155107 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-data\") pod \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " Oct 30 17:28:23.155273 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.155176 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-token\" (UniqueName: \"kubernetes.io/projected/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-token\") pod \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " Oct 30 17:28:23.155273 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.155205 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-kube-api-access-9vrdt\" (UniqueName: \"kubernetes.io/projected/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-kube-api-access-9vrdt\") pod \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " Oct 30 17:28:23.155273 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.155239 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-workload-certs\") pod \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\" (UID: \"fe89fc65-9620-4dc6-98da-cd1a3d57dbc3\") " Oct 30 17:28:23.156083 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.155833 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-credential-socket" pod "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" (UID: "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:23.156083 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.156061 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istiod-ca-cert" pod "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" (UID: "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:28:23.156318 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.156293 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-workload-socket" pod "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" (UID: "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:23.156392 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.156366 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-workload-certs" pod "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" (UID: "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:23.156861 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.156830 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-data" pod "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" (UID: "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:23.158246 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.158223 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-kube-api-access-9vrdt" pod "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" (UID: "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3"). InnerVolumeSpecName "kube-api-access-9vrdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:23.158338 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.158253 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-podinfo" pod "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" (UID: "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:28:23.158448 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.158430 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-envoy" pod "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" (UID: "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:23.158448 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.158435 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-token" pod "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" (UID: "fe89fc65-9620-4dc6-98da-cd1a3d57dbc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:23.256558 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.256515 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vrdt\" (UniqueName: \"kubernetes.io/projected/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-kube-api-access-9vrdt\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:23.256558 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.256552 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:23.256558 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.256563 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:23.256957 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.256571 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:23.256957 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.256579 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:23.256957 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.256588 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:23.256957 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.256596 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:23.256957 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.256604 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:23.256957 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:23.256617 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:24.056458 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:24.056426 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7" Oct 30 17:28:24.158532 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:24.158458 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7"] Oct 30 17:28:24.161655 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:24.161626 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f3503b-openshift-default-668846555f-64rg7"] Oct 30 17:28:25.082872 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:25.082821 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" podUID="ce164706-e748-479e-a1a6-7977e983bc9e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:28:25.083330 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:25.082844 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" podUID="40c65962-30bc-4ea9-b938-4a4917e875cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:28:25.808423 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:25.808390 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" path="/var/lib/kubelet/pods/fe89fc65-9620-4dc6-98da-cd1a3d57dbc3/volumes" Oct 30 17:28:35.082734 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:35.082681 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" podUID="40c65962-30bc-4ea9-b938-4a4917e875cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:28:35.082734 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:35.082687 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" podUID="ce164706-e748-479e-a1a6-7977e983bc9e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:28:38.842640 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:38.842603 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj"] Oct 30 17:28:39.582198 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:39.582157 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8"] Oct 30 17:28:39.582411 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:39.582380 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" podUID="1ad5ad27-a300-44f3-80af-ac24aabf8ebd" containerName="maas-api" containerID="cri-o://0b80fe178f7223ae71baa94622e6726fbb8e46df2b175988f094fb30007d9239" gracePeriod=30 Oct 30 17:28:39.851628 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:39.851594 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 17:28:39.928881 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:39.928846 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-openshift-service-ca\") pod \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\" (UID: \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\") " Oct 30 17:28:39.929085 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:39.928941 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-metrics-certs\") pod \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\" (UID: \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\") " Oct 30 17:28:39.929085 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:39.928998 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-maas-api-tls\") pod \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\" (UID: \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\") " Oct 30 17:28:39.929248 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:39.929114 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-kube-api-access-lnpsv\" (UniqueName: \"kubernetes.io/projected/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-kube-api-access-lnpsv\") pod \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\" (UID: \"1ad5ad27-a300-44f3-80af-ac24aabf8ebd\") " Oct 30 17:28:39.929310 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:39.929255 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-openshift-service-ca" pod "1ad5ad27-a300-44f3-80af-ac24aabf8ebd" (UID: "1ad5ad27-a300-44f3-80af-ac24aabf8ebd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:28:39.929538 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:39.929517 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:39.931307 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:39.931284 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-metrics-certs" pod "1ad5ad27-a300-44f3-80af-ac24aabf8ebd" (UID: "1ad5ad27-a300-44f3-80af-ac24aabf8ebd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:28:39.931413 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:39.931332 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-maas-api-tls" pod "1ad5ad27-a300-44f3-80af-ac24aabf8ebd" (UID: "1ad5ad27-a300-44f3-80af-ac24aabf8ebd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:28:39.931413 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:39.931345 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-kube-api-access-lnpsv" pod "1ad5ad27-a300-44f3-80af-ac24aabf8ebd" (UID: "1ad5ad27-a300-44f3-80af-ac24aabf8ebd"). InnerVolumeSpecName "kube-api-access-lnpsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:40.030196 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:40.030159 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnpsv\" (UniqueName: \"kubernetes.io/projected/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-kube-api-access-lnpsv\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:40.030196 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:40.030191 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:40.030196 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:40.030200 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad5ad27-a300-44f3-80af-ac24aabf8ebd-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:40.155926 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:40.155834 4861 generic.go:356] "Generic (PLEG): container finished" podID="1ad5ad27-a300-44f3-80af-ac24aabf8ebd" containerID="0b80fe178f7223ae71baa94622e6726fbb8e46df2b175988f094fb30007d9239" exitCode=0 Oct 30 17:28:40.155926 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:40.155895 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" Oct 30 17:28:40.156121 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:40.155922 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" event={"ID":"1ad5ad27-a300-44f3-80af-ac24aabf8ebd","Type":"ContainerDied","Data":"0b80fe178f7223ae71baa94622e6726fbb8e46df2b175988f094fb30007d9239"} Oct 30 17:28:40.156121 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:40.155997 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8" event={"ID":"1ad5ad27-a300-44f3-80af-ac24aabf8ebd","Type":"ContainerDied","Data":"038ac54b49b231429855e10d33c8911d46eea0ae79ff0deac36ac596a6a332c9"} Oct 30 17:28:40.156121 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:40.156028 4861 scope.go:122] "RemoveContainer" containerID="0b80fe178f7223ae71baa94622e6726fbb8e46df2b175988f094fb30007d9239" Oct 30 17:28:40.166180 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:40.166164 4861 scope.go:122] "RemoveContainer" containerID="0b80fe178f7223ae71baa94622e6726fbb8e46df2b175988f094fb30007d9239" Oct 30 17:28:40.166476 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:28:40.166455 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b80fe178f7223ae71baa94622e6726fbb8e46df2b175988f094fb30007d9239\": container with ID starting with 0b80fe178f7223ae71baa94622e6726fbb8e46df2b175988f094fb30007d9239 not found: ID does not exist" containerID="0b80fe178f7223ae71baa94622e6726fbb8e46df2b175988f094fb30007d9239" Oct 30 17:28:40.166527 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:40.166482 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b80fe178f7223ae71baa94622e6726fbb8e46df2b175988f094fb30007d9239"} err="failed to get container status \"0b80fe178f7223ae71baa94622e6726fbb8e46df2b175988f094fb30007d9239\": rpc error: code = NotFound desc = could not find container \"0b80fe178f7223ae71baa94622e6726fbb8e46df2b175988f094fb30007d9239\": container with ID starting with 0b80fe178f7223ae71baa94622e6726fbb8e46df2b175988f094fb30007d9239 not found: ID does not exist" Oct 30 17:28:40.214306 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:40.214282 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8"] Oct 30 17:28:40.227704 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:40.227684 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-cf4621-85ff4d7fd9-4l7f8"] Oct 30 17:28:41.806085 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:41.806052 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ad5ad27-a300-44f3-80af-ac24aabf8ebd" path="/var/lib/kubelet/pods/1ad5ad27-a300-44f3-80af-ac24aabf8ebd/volumes" Oct 30 17:28:43.181328 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.181107 4861 generic.go:356] "Generic (PLEG): container finished" podID="ce164706-e748-479e-a1a6-7977e983bc9e" containerID="81b274cea3578b2b0f44f15cb1034c7ad6fba6244210eb9e19b3701bd7a96f9f" exitCode=0 Oct 30 17:28:43.181328 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.181267 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" event={"ID":"ce164706-e748-479e-a1a6-7977e983bc9e","Type":"ContainerDied","Data":"81b274cea3578b2b0f44f15cb1034c7ad6fba6244210eb9e19b3701bd7a96f9f"} Oct 30 17:28:43.184014 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.183985 4861 generic.go:356] "Generic (PLEG): container finished" podID="40c65962-30bc-4ea9-b938-4a4917e875cc" containerID="87eccaefa07e400cd38b7ac2c6833f5c30115424fddc38a412fe17abfb84133f" exitCode=0 Oct 30 17:28:43.184165 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.184045 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" event={"ID":"40c65962-30bc-4ea9-b938-4a4917e875cc","Type":"ContainerDied","Data":"87eccaefa07e400cd38b7ac2c6833f5c30115424fddc38a412fe17abfb84133f"} Oct 30 17:28:43.245946 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.245914 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" Oct 30 17:28:43.268426 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.268402 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" Oct 30 17:28:43.360021 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.359989 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c65962-30bc-4ea9-b938-4a4917e875cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c65962-30bc-4ea9-b938-4a4917e875cc-plugins-config-volume\") pod \"40c65962-30bc-4ea9-b938-4a4917e875cc\" (UID: \"40c65962-30bc-4ea9-b938-4a4917e875cc\") " Oct 30 17:28:43.360192 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.360038 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce164706-e748-479e-a1a6-7977e983bc9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce164706-e748-479e-a1a6-7977e983bc9e-plugins-config-volume\") pod \"ce164706-e748-479e-a1a6-7977e983bc9e\" (UID: \"ce164706-e748-479e-a1a6-7977e983bc9e\") " Oct 30 17:28:43.360192 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.360084 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce164706-e748-479e-a1a6-7977e983bc9e-kube-api-access-d9jr5\" (UniqueName: \"kubernetes.io/projected/ce164706-e748-479e-a1a6-7977e983bc9e-kube-api-access-d9jr5\") pod \"ce164706-e748-479e-a1a6-7977e983bc9e\" (UID: \"ce164706-e748-479e-a1a6-7977e983bc9e\") " Oct 30 17:28:43.360293 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.360192 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c65962-30bc-4ea9-b938-4a4917e875cc-kube-api-access-qswrn\" (UniqueName: \"kubernetes.io/projected/40c65962-30bc-4ea9-b938-4a4917e875cc-kube-api-access-qswrn\") pod \"40c65962-30bc-4ea9-b938-4a4917e875cc\" (UID: \"40c65962-30bc-4ea9-b938-4a4917e875cc\") " Oct 30 17:28:43.360464 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.360440 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c65962-30bc-4ea9-b938-4a4917e875cc-plugins-config-volume" pod "40c65962-30bc-4ea9-b938-4a4917e875cc" (UID: "40c65962-30bc-4ea9-b938-4a4917e875cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:28:43.360464 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.360443 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce164706-e748-479e-a1a6-7977e983bc9e-plugins-config-volume" pod "ce164706-e748-479e-a1a6-7977e983bc9e" (UID: "ce164706-e748-479e-a1a6-7977e983bc9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:28:43.360617 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.360578 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce164706-e748-479e-a1a6-7977e983bc9e-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:43.360617 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.360591 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c65962-30bc-4ea9-b938-4a4917e875cc-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:43.362560 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.362536 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c65962-30bc-4ea9-b938-4a4917e875cc-kube-api-access-qswrn" pod "40c65962-30bc-4ea9-b938-4a4917e875cc" (UID: "40c65962-30bc-4ea9-b938-4a4917e875cc"). InnerVolumeSpecName "kube-api-access-qswrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:43.362651 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.362610 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce164706-e748-479e-a1a6-7977e983bc9e-kube-api-access-d9jr5" pod "ce164706-e748-479e-a1a6-7977e983bc9e" (UID: "ce164706-e748-479e-a1a6-7977e983bc9e"). InnerVolumeSpecName "kube-api-access-d9jr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:43.461784 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.461757 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qswrn\" (UniqueName: \"kubernetes.io/projected/40c65962-30bc-4ea9-b938-4a4917e875cc-kube-api-access-qswrn\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:43.461784 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.461780 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9jr5\" (UniqueName: \"kubernetes.io/projected/ce164706-e748-479e-a1a6-7977e983bc9e-kube-api-access-d9jr5\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:43.843952 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:43.843884 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" podUID="bb5c8a89-0cb6-4225-97c9-189984852da9" containerName="payload-pre-processing" containerID="cri-o://cb25c8018e819a603966a3be766e08561e425f90d830120d13ad795ff13280ae" gracePeriod=25 Oct 30 17:28:44.190302 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:44.190270 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" Oct 30 17:28:44.190821 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:44.190299 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj" event={"ID":"40c65962-30bc-4ea9-b938-4a4917e875cc","Type":"ContainerDied","Data":"5b17327c9c6f8a9d9d3a40cfbc7ee465d3076c562627a61fd7b181668034f809"} Oct 30 17:28:44.190821 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:44.190363 4861 scope.go:122] "RemoveContainer" containerID="87eccaefa07e400cd38b7ac2c6833f5c30115424fddc38a412fe17abfb84133f" Oct 30 17:28:44.191750 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:44.191707 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" event={"ID":"ce164706-e748-479e-a1a6-7977e983bc9e","Type":"ContainerDied","Data":"4b18ac22e4fd89fc4df85323b23c4fa581861426fabbf3372481a9242eaafa88"} Oct 30 17:28:44.191844 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:44.191757 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp" Oct 30 17:28:44.199911 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:44.199896 4861 scope.go:122] "RemoveContainer" containerID="81b274cea3578b2b0f44f15cb1034c7ad6fba6244210eb9e19b3701bd7a96f9f" Oct 30 17:28:44.246060 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:44.246038 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj"] Oct 30 17:28:44.259652 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:44.259631 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-13c495-cb894464c-lnwdj"] Oct 30 17:28:44.295679 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:44.295650 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp"] Oct 30 17:28:44.302533 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:44.302513 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-13c495-66857b595b-zb6bp"] Oct 30 17:28:44.643768 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:44.643700 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" podUID="bb5c8a89-0cb6-4225-97c9-189984852da9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:28:45.805969 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:45.805934 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c65962-30bc-4ea9-b938-4a4917e875cc" path="/var/lib/kubelet/pods/40c65962-30bc-4ea9-b938-4a4917e875cc/volumes" Oct 30 17:28:45.806348 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:45.806319 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce164706-e748-479e-a1a6-7977e983bc9e" path="/var/lib/kubelet/pods/ce164706-e748-479e-a1a6-7977e983bc9e/volumes" Oct 30 17:28:47.976021 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:47.975985 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd"] Oct 30 17:28:47.976512 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:47.976271 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" podUID="41e4e14f-88e4-4fac-a6fd-04baae40e066" containerName="istio-proxy" containerID="cri-o://17a2de0fa38cdd7949ff8caa121b3f3823de4b5db372f05d9b95d629d26d5db5" gracePeriod=30 Oct 30 17:28:50.289240 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:28:50.289205 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:28:53.105389 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.105361 4861 scope.go:122] "RemoveContainer" containerID="0ed9b3b072fac4b32f428cacde8c13d2eaa86031dbe367cb636569ef104fedcf" Oct 30 17:28:53.119315 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.119211 4861 scope.go:122] "RemoveContainer" containerID="9db8cf48652149075be5468639cbbb7f230e0b57b594b8ca7ffe649b13560d8c" Oct 30 17:28:53.242957 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.242887 4861 generic.go:356] "Generic (PLEG): container finished" podID="41e4e14f-88e4-4fac-a6fd-04baae40e066" containerID="17a2de0fa38cdd7949ff8caa121b3f3823de4b5db372f05d9b95d629d26d5db5" exitCode=0 Oct 30 17:28:53.243065 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.242959 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" event={"ID":"41e4e14f-88e4-4fac-a6fd-04baae40e066","Type":"ContainerDied","Data":"17a2de0fa38cdd7949ff8caa121b3f3823de4b5db372f05d9b95d629d26d5db5"} Oct 30 17:28:53.243065 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.243053 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" event={"ID":"41e4e14f-88e4-4fac-a6fd-04baae40e066","Type":"ContainerDied","Data":"28db47c3f749ad50c4f85da15abab01138ea22e83439c3bba8015bef2599e911"} Oct 30 17:28:53.243150 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.243071 4861 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28db47c3f749ad50c4f85da15abab01138ea22e83439c3bba8015bef2599e911" Oct 30 17:28:53.244062 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.244043 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 17:28:53.357167 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.357138 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e4e14f-88e4-4fac-a6fd-04baae40e066-kube-api-access-4lw78\" (UniqueName: \"kubernetes.io/projected/41e4e14f-88e4-4fac-a6fd-04baae40e066-kube-api-access-4lw78\") pod \"41e4e14f-88e4-4fac-a6fd-04baae40e066\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " Oct 30 17:28:53.357360 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.357186 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-podinfo\") pod \"41e4e14f-88e4-4fac-a6fd-04baae40e066\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " Oct 30 17:28:53.357360 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.357206 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e4e14f-88e4-4fac-a6fd-04baae40e066-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e4e14f-88e4-4fac-a6fd-04baae40e066-istiod-ca-cert\") pod \"41e4e14f-88e4-4fac-a6fd-04baae40e066\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " Oct 30 17:28:53.357360 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.357240 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-workload-socket\") pod \"41e4e14f-88e4-4fac-a6fd-04baae40e066\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " Oct 30 17:28:53.357360 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.357308 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-workload-certs\") pod \"41e4e14f-88e4-4fac-a6fd-04baae40e066\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " Oct 30 17:28:53.357360 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.357354 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-data\") pod \"41e4e14f-88e4-4fac-a6fd-04baae40e066\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " Oct 30 17:28:53.357693 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.357377 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-credential-socket\") pod \"41e4e14f-88e4-4fac-a6fd-04baae40e066\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " Oct 30 17:28:53.357693 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.357428 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-token\" (UniqueName: \"kubernetes.io/projected/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-token\") pod \"41e4e14f-88e4-4fac-a6fd-04baae40e066\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " Oct 30 17:28:53.357693 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.357509 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-envoy\") pod \"41e4e14f-88e4-4fac-a6fd-04baae40e066\" (UID: \"41e4e14f-88e4-4fac-a6fd-04baae40e066\") " Oct 30 17:28:53.357693 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.357632 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-workload-socket" pod "41e4e14f-88e4-4fac-a6fd-04baae40e066" (UID: "41e4e14f-88e4-4fac-a6fd-04baae40e066"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:53.357693 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.357676 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-data" pod "41e4e14f-88e4-4fac-a6fd-04baae40e066" (UID: "41e4e14f-88e4-4fac-a6fd-04baae40e066"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:53.357693 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.357677 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-workload-certs" pod "41e4e14f-88e4-4fac-a6fd-04baae40e066" (UID: "41e4e14f-88e4-4fac-a6fd-04baae40e066"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:53.358095 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.357702 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e4e14f-88e4-4fac-a6fd-04baae40e066-istiod-ca-cert" pod "41e4e14f-88e4-4fac-a6fd-04baae40e066" (UID: "41e4e14f-88e4-4fac-a6fd-04baae40e066"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:28:53.358095 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.357955 4861 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-data\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:53.358095 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.357981 4861 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41e4e14f-88e4-4fac-a6fd-04baae40e066-istiod-ca-cert\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:53.358095 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.357997 4861 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-workload-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:53.358095 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.358012 4861 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-workload-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:53.358472 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.358422 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-credential-socket" pod "41e4e14f-88e4-4fac-a6fd-04baae40e066" (UID: "41e4e14f-88e4-4fac-a6fd-04baae40e066"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:53.359705 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.359676 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e4e14f-88e4-4fac-a6fd-04baae40e066-kube-api-access-4lw78" pod "41e4e14f-88e4-4fac-a6fd-04baae40e066" (UID: "41e4e14f-88e4-4fac-a6fd-04baae40e066"). InnerVolumeSpecName "kube-api-access-4lw78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:53.359705 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.359695 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-podinfo" pod "41e4e14f-88e4-4fac-a6fd-04baae40e066" (UID: "41e4e14f-88e4-4fac-a6fd-04baae40e066"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:28:53.359892 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.359846 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-envoy" pod "41e4e14f-88e4-4fac-a6fd-04baae40e066" (UID: "41e4e14f-88e4-4fac-a6fd-04baae40e066"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:28:53.360075 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.360052 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-token" pod "41e4e14f-88e4-4fac-a6fd-04baae40e066" (UID: "41e4e14f-88e4-4fac-a6fd-04baae40e066"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:28:53.459381 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.459345 4861 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-envoy\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:53.459381 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.459374 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lw78\" (UniqueName: \"kubernetes.io/projected/41e4e14f-88e4-4fac-a6fd-04baae40e066-kube-api-access-4lw78\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:53.459381 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.459385 4861 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-podinfo\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:53.459644 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.459393 4861 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41e4e14f-88e4-4fac-a6fd-04baae40e066-credential-socket\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:53.459644 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:53.459402 4861 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41e4e14f-88e4-4fac-a6fd-04baae40e066-istio-token\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:28:54.247911 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:54.247879 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd" Oct 30 17:28:54.319473 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:54.319429 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd"] Oct 30 17:28:54.332689 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:54.332655 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cf4621-openshift-default-866b48dc69-6rrpd"] Oct 30 17:28:54.643348 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:54.643262 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" podUID="bb5c8a89-0cb6-4225-97c9-189984852da9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:28:55.805876 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:55.805840 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e4e14f-88e4-4fac-a6fd-04baae40e066" path="/var/lib/kubelet/pods/41e4e14f-88e4-4fac-a6fd-04baae40e066/volumes" Oct 30 17:28:59.059778 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:59.059736 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts"] Oct 30 17:28:59.091894 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:28:59.091863 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq"] Oct 30 17:29:04.060577 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.060525 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" podUID="3b0f1529-cc32-4603-987f-55b7d3a54c2a" containerName="payload-processing" containerID="cri-o://991db04ba391e2a05aa5a48536271ebebc7b731dbf3f07849783c71eb49dff6f" gracePeriod=25 Oct 30 17:29:04.092200 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.092164 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" podUID="199a5f3f-241a-4d65-813c-928af105ac2b" containerName="payload-pre-processing" containerID="cri-o://959e2dee1c995a8b124138f3c469ca9529c0f40e745f2e95596293b224fa48c2" gracePeriod=25 Oct 30 17:29:04.113187 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.113116 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" Oct 30 17:29:04.270963 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.270938 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5c8a89-0cb6-4225-97c9-189984852da9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb5c8a89-0cb6-4225-97c9-189984852da9-plugins-config-volume\") pod \"bb5c8a89-0cb6-4225-97c9-189984852da9\" (UID: \"bb5c8a89-0cb6-4225-97c9-189984852da9\") " Oct 30 17:29:04.271143 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.270970 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5c8a89-0cb6-4225-97c9-189984852da9-kube-api-access-9lcgm\" (UniqueName: \"kubernetes.io/projected/bb5c8a89-0cb6-4225-97c9-189984852da9-kube-api-access-9lcgm\") pod \"bb5c8a89-0cb6-4225-97c9-189984852da9\" (UID: \"bb5c8a89-0cb6-4225-97c9-189984852da9\") " Oct 30 17:29:04.271331 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.271304 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5c8a89-0cb6-4225-97c9-189984852da9-plugins-config-volume" pod "bb5c8a89-0cb6-4225-97c9-189984852da9" (UID: "bb5c8a89-0cb6-4225-97c9-189984852da9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:04.273372 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.273350 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5c8a89-0cb6-4225-97c9-189984852da9-kube-api-access-9lcgm" pod "bb5c8a89-0cb6-4225-97c9-189984852da9" (UID: "bb5c8a89-0cb6-4225-97c9-189984852da9"). InnerVolumeSpecName "kube-api-access-9lcgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:04.303973 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.303948 4861 generic.go:356] "Generic (PLEG): container finished" podID="bb5c8a89-0cb6-4225-97c9-189984852da9" containerID="cb25c8018e819a603966a3be766e08561e425f90d830120d13ad795ff13280ae" exitCode=0 Oct 30 17:29:04.304098 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.304017 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" Oct 30 17:29:04.304165 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.304023 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" event={"ID":"bb5c8a89-0cb6-4225-97c9-189984852da9","Type":"ContainerDied","Data":"cb25c8018e819a603966a3be766e08561e425f90d830120d13ad795ff13280ae"} Oct 30 17:29:04.304165 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.304135 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj" event={"ID":"bb5c8a89-0cb6-4225-97c9-189984852da9","Type":"ContainerDied","Data":"9ab4daa57e3f93a283674b86239cac176e13a8fca29321ed58214096c833a8e1"} Oct 30 17:29:04.304165 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.304157 4861 scope.go:122] "RemoveContainer" containerID="cb25c8018e819a603966a3be766e08561e425f90d830120d13ad795ff13280ae" Oct 30 17:29:04.313901 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.313885 4861 scope.go:122] "RemoveContainer" containerID="cb25c8018e819a603966a3be766e08561e425f90d830120d13ad795ff13280ae" Oct 30 17:29:04.314164 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:29:04.314148 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb25c8018e819a603966a3be766e08561e425f90d830120d13ad795ff13280ae\": container with ID starting with cb25c8018e819a603966a3be766e08561e425f90d830120d13ad795ff13280ae not found: ID does not exist" containerID="cb25c8018e819a603966a3be766e08561e425f90d830120d13ad795ff13280ae" Oct 30 17:29:04.314227 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.314169 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb25c8018e819a603966a3be766e08561e425f90d830120d13ad795ff13280ae"} err="failed to get container status \"cb25c8018e819a603966a3be766e08561e425f90d830120d13ad795ff13280ae\": rpc error: code = NotFound desc = could not find container \"cb25c8018e819a603966a3be766e08561e425f90d830120d13ad795ff13280ae\": container with ID starting with cb25c8018e819a603966a3be766e08561e425f90d830120d13ad795ff13280ae not found: ID does not exist" Oct 30 17:29:04.362001 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.361973 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj"] Oct 30 17:29:04.366838 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.366789 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-cf4621-69594599cf-2h7mj"] Oct 30 17:29:04.372416 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.372397 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb5c8a89-0cb6-4225-97c9-189984852da9-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:29:04.372468 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:04.372419 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lcgm\" (UniqueName: \"kubernetes.io/projected/bb5c8a89-0cb6-4225-97c9-189984852da9-kube-api-access-9lcgm\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:29:05.809808 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:05.809761 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5c8a89-0cb6-4225-97c9-189984852da9" path="/var/lib/kubelet/pods/bb5c8a89-0cb6-4225-97c9-189984852da9/volumes" Oct 30 17:29:09.864692 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:09.864650 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" podUID="3b0f1529-cc32-4603-987f-55b7d3a54c2a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:29:09.865092 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:09.864701 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" podUID="199a5f3f-241a-4d65-813c-928af105ac2b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:29:19.865342 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:19.865301 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" podUID="199a5f3f-241a-4d65-813c-928af105ac2b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:29:19.865787 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:19.865301 4861 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" podUID="3b0f1529-cc32-4603-987f-55b7d3a54c2a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:29:20.290484 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:29:20.290446 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:29:24.337504 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.337480 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" Oct 30 17:29:24.363131 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.363109 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" Oct 30 17:29:24.404446 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.404395 4861 generic.go:356] "Generic (PLEG): container finished" podID="3b0f1529-cc32-4603-987f-55b7d3a54c2a" containerID="991db04ba391e2a05aa5a48536271ebebc7b731dbf3f07849783c71eb49dff6f" exitCode=0 Oct 30 17:29:24.404627 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.404449 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" event={"ID":"3b0f1529-cc32-4603-987f-55b7d3a54c2a","Type":"ContainerDied","Data":"991db04ba391e2a05aa5a48536271ebebc7b731dbf3f07849783c71eb49dff6f"} Oct 30 17:29:24.404627 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.404498 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" event={"ID":"3b0f1529-cc32-4603-987f-55b7d3a54c2a","Type":"ContainerDied","Data":"1da294cedef5b7e64027bceaf564e676d0a9831096f0d23fce43e3d0a5f387ae"} Oct 30 17:29:24.404627 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.404432 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts" Oct 30 17:29:24.404627 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.404526 4861 scope.go:122] "RemoveContainer" containerID="991db04ba391e2a05aa5a48536271ebebc7b731dbf3f07849783c71eb49dff6f" Oct 30 17:29:24.407268 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.407239 4861 generic.go:356] "Generic (PLEG): container finished" podID="199a5f3f-241a-4d65-813c-928af105ac2b" containerID="959e2dee1c995a8b124138f3c469ca9529c0f40e745f2e95596293b224fa48c2" exitCode=0 Oct 30 17:29:24.407385 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.407292 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" Oct 30 17:29:24.407385 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.407335 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" event={"ID":"199a5f3f-241a-4d65-813c-928af105ac2b","Type":"ContainerDied","Data":"959e2dee1c995a8b124138f3c469ca9529c0f40e745f2e95596293b224fa48c2"} Oct 30 17:29:24.407385 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.407375 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq" event={"ID":"199a5f3f-241a-4d65-813c-928af105ac2b","Type":"ContainerDied","Data":"e425bef0a39266ecd524cfb11821306efa18ba23454af84a473837c3ff80c930"} Oct 30 17:29:24.417858 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.417835 4861 scope.go:122] "RemoveContainer" containerID="991db04ba391e2a05aa5a48536271ebebc7b731dbf3f07849783c71eb49dff6f" Oct 30 17:29:24.418136 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:29:24.418119 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991db04ba391e2a05aa5a48536271ebebc7b731dbf3f07849783c71eb49dff6f\": container with ID starting with 991db04ba391e2a05aa5a48536271ebebc7b731dbf3f07849783c71eb49dff6f not found: ID does not exist" containerID="991db04ba391e2a05aa5a48536271ebebc7b731dbf3f07849783c71eb49dff6f" Oct 30 17:29:24.418193 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.418145 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991db04ba391e2a05aa5a48536271ebebc7b731dbf3f07849783c71eb49dff6f"} err="failed to get container status \"991db04ba391e2a05aa5a48536271ebebc7b731dbf3f07849783c71eb49dff6f\": rpc error: code = NotFound desc = could not find container \"991db04ba391e2a05aa5a48536271ebebc7b731dbf3f07849783c71eb49dff6f\": container with ID starting with 991db04ba391e2a05aa5a48536271ebebc7b731dbf3f07849783c71eb49dff6f not found: ID does not exist" Oct 30 17:29:24.418193 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.418161 4861 scope.go:122] "RemoveContainer" containerID="959e2dee1c995a8b124138f3c469ca9529c0f40e745f2e95596293b224fa48c2" Oct 30 17:29:24.428286 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.428270 4861 scope.go:122] "RemoveContainer" containerID="959e2dee1c995a8b124138f3c469ca9529c0f40e745f2e95596293b224fa48c2" Oct 30 17:29:24.428581 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:29:24.428561 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"959e2dee1c995a8b124138f3c469ca9529c0f40e745f2e95596293b224fa48c2\": container with ID starting with 959e2dee1c995a8b124138f3c469ca9529c0f40e745f2e95596293b224fa48c2 not found: ID does not exist" containerID="959e2dee1c995a8b124138f3c469ca9529c0f40e745f2e95596293b224fa48c2" Oct 30 17:29:24.428648 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.428588 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"959e2dee1c995a8b124138f3c469ca9529c0f40e745f2e95596293b224fa48c2"} err="failed to get container status \"959e2dee1c995a8b124138f3c469ca9529c0f40e745f2e95596293b224fa48c2\": rpc error: code = NotFound desc = could not find container \"959e2dee1c995a8b124138f3c469ca9529c0f40e745f2e95596293b224fa48c2\": container with ID starting with 959e2dee1c995a8b124138f3c469ca9529c0f40e745f2e95596293b224fa48c2 not found: ID does not exist" Oct 30 17:29:24.452954 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.452926 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0f1529-cc32-4603-987f-55b7d3a54c2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0f1529-cc32-4603-987f-55b7d3a54c2a-plugins-config-volume\") pod \"3b0f1529-cc32-4603-987f-55b7d3a54c2a\" (UID: \"3b0f1529-cc32-4603-987f-55b7d3a54c2a\") " Oct 30 17:29:24.453065 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.452979 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/199a5f3f-241a-4d65-813c-928af105ac2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199a5f3f-241a-4d65-813c-928af105ac2b-plugins-config-volume\") pod \"199a5f3f-241a-4d65-813c-928af105ac2b\" (UID: \"199a5f3f-241a-4d65-813c-928af105ac2b\") " Oct 30 17:29:24.453065 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.453042 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/199a5f3f-241a-4d65-813c-928af105ac2b-kube-api-access-wmhzq\" (UniqueName: \"kubernetes.io/projected/199a5f3f-241a-4d65-813c-928af105ac2b-kube-api-access-wmhzq\") pod \"199a5f3f-241a-4d65-813c-928af105ac2b\" (UID: \"199a5f3f-241a-4d65-813c-928af105ac2b\") " Oct 30 17:29:24.453240 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.453087 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0f1529-cc32-4603-987f-55b7d3a54c2a-kube-api-access-rr97j\" (UniqueName: \"kubernetes.io/projected/3b0f1529-cc32-4603-987f-55b7d3a54c2a-kube-api-access-rr97j\") pod \"3b0f1529-cc32-4603-987f-55b7d3a54c2a\" (UID: \"3b0f1529-cc32-4603-987f-55b7d3a54c2a\") " Oct 30 17:29:24.453321 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.453296 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0f1529-cc32-4603-987f-55b7d3a54c2a-plugins-config-volume" pod "3b0f1529-cc32-4603-987f-55b7d3a54c2a" (UID: "3b0f1529-cc32-4603-987f-55b7d3a54c2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:24.453374 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.453328 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199a5f3f-241a-4d65-813c-928af105ac2b-plugins-config-volume" pod "199a5f3f-241a-4d65-813c-928af105ac2b" (UID: "199a5f3f-241a-4d65-813c-928af105ac2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:24.453473 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.453450 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0f1529-cc32-4603-987f-55b7d3a54c2a-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:29:24.453473 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.453471 4861 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199a5f3f-241a-4d65-813c-928af105ac2b-plugins-config-volume\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:29:24.455246 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.455227 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199a5f3f-241a-4d65-813c-928af105ac2b-kube-api-access-wmhzq" pod "199a5f3f-241a-4d65-813c-928af105ac2b" (UID: "199a5f3f-241a-4d65-813c-928af105ac2b"). InnerVolumeSpecName "kube-api-access-wmhzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:24.455327 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.455312 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0f1529-cc32-4603-987f-55b7d3a54c2a-kube-api-access-rr97j" pod "3b0f1529-cc32-4603-987f-55b7d3a54c2a" (UID: "3b0f1529-cc32-4603-987f-55b7d3a54c2a"). InnerVolumeSpecName "kube-api-access-rr97j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:24.554713 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.554675 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr97j\" (UniqueName: \"kubernetes.io/projected/3b0f1529-cc32-4603-987f-55b7d3a54c2a-kube-api-access-rr97j\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:29:24.554713 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.554708 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmhzq\" (UniqueName: \"kubernetes.io/projected/199a5f3f-241a-4d65-813c-928af105ac2b-kube-api-access-wmhzq\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:29:24.758452 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.758425 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts"] Oct 30 17:29:24.762497 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.762472 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-3ca339-5fcc778c56-2vpts"] Oct 30 17:29:24.799230 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.799198 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq"] Oct 30 17:29:24.805338 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:24.805311 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-3ca339-5fc569779f-x6vcq"] Oct 30 17:29:25.804986 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:25.804950 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="199a5f3f-241a-4d65-813c-928af105ac2b" path="/var/lib/kubelet/pods/199a5f3f-241a-4d65-813c-928af105ac2b/volumes" Oct 30 17:29:25.805376 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:25.805312 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0f1529-cc32-4603-987f-55b7d3a54c2a" path="/var/lib/kubelet/pods/3b0f1529-cc32-4603-987f-55b7d3a54c2a/volumes" Oct 30 17:29:36.647842 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:36.647704 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb"] Oct 30 17:29:36.648330 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:36.647934 4861 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" podUID="1d6544b2-2ae4-4901-a710-f37d04215115" containerName="maas-api" containerID="cri-o://ef674785f9a9edd66d853f135dcbd7fab1aa5e84a651b76b8903ed1d3cb93281" gracePeriod=30 Oct 30 17:29:36.911860 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:36.911832 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 17:29:36.959435 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:36.959396 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d6544b2-2ae4-4901-a710-f37d04215115-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d6544b2-2ae4-4901-a710-f37d04215115-maas-api-tls\") pod \"1d6544b2-2ae4-4901-a710-f37d04215115\" (UID: \"1d6544b2-2ae4-4901-a710-f37d04215115\") " Oct 30 17:29:36.959617 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:36.959547 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d6544b2-2ae4-4901-a710-f37d04215115-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6544b2-2ae4-4901-a710-f37d04215115-openshift-service-ca\") pod \"1d6544b2-2ae4-4901-a710-f37d04215115\" (UID: \"1d6544b2-2ae4-4901-a710-f37d04215115\") " Oct 30 17:29:36.959617 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:36.959614 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d6544b2-2ae4-4901-a710-f37d04215115-kube-api-access-cvmr8\" (UniqueName: \"kubernetes.io/projected/1d6544b2-2ae4-4901-a710-f37d04215115-kube-api-access-cvmr8\") pod \"1d6544b2-2ae4-4901-a710-f37d04215115\" (UID: \"1d6544b2-2ae4-4901-a710-f37d04215115\") " Oct 30 17:29:36.959791 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:36.959683 4861 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d6544b2-2ae4-4901-a710-f37d04215115-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d6544b2-2ae4-4901-a710-f37d04215115-metrics-certs\") pod \"1d6544b2-2ae4-4901-a710-f37d04215115\" (UID: \"1d6544b2-2ae4-4901-a710-f37d04215115\") " Oct 30 17:29:36.960134 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:36.960097 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d6544b2-2ae4-4901-a710-f37d04215115-openshift-service-ca" pod "1d6544b2-2ae4-4901-a710-f37d04215115" (UID: "1d6544b2-2ae4-4901-a710-f37d04215115"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:29:36.962306 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:36.962277 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d6544b2-2ae4-4901-a710-f37d04215115-maas-api-tls" pod "1d6544b2-2ae4-4901-a710-f37d04215115" (UID: "1d6544b2-2ae4-4901-a710-f37d04215115"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:29:36.963083 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:36.963055 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d6544b2-2ae4-4901-a710-f37d04215115-kube-api-access-cvmr8" pod "1d6544b2-2ae4-4901-a710-f37d04215115" (UID: "1d6544b2-2ae4-4901-a710-f37d04215115"). InnerVolumeSpecName "kube-api-access-cvmr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:36.963183 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:36.963125 4861 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d6544b2-2ae4-4901-a710-f37d04215115-metrics-certs" pod "1d6544b2-2ae4-4901-a710-f37d04215115" (UID: "1d6544b2-2ae4-4901-a710-f37d04215115"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:29:37.060521 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:37.060487 4861 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d6544b2-2ae4-4901-a710-f37d04215115-metrics-certs\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:29:37.060521 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:37.060516 4861 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d6544b2-2ae4-4901-a710-f37d04215115-maas-api-tls\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:29:37.060702 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:37.060532 4861 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d6544b2-2ae4-4901-a710-f37d04215115-openshift-service-ca\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:29:37.060702 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:37.060544 4861 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvmr8\" (UniqueName: \"kubernetes.io/projected/1d6544b2-2ae4-4901-a710-f37d04215115-kube-api-access-cvmr8\") on node \"ip-10-0-131-179.ec2.internal\" DevicePath \"\"" Oct 30 17:29:37.479234 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:37.479197 4861 generic.go:356] "Generic (PLEG): container finished" podID="1d6544b2-2ae4-4901-a710-f37d04215115" containerID="ef674785f9a9edd66d853f135dcbd7fab1aa5e84a651b76b8903ed1d3cb93281" exitCode=0 Oct 30 17:29:37.479420 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:37.479250 4861 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" Oct 30 17:29:37.479420 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:37.479288 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" event={"ID":"1d6544b2-2ae4-4901-a710-f37d04215115","Type":"ContainerDied","Data":"ef674785f9a9edd66d853f135dcbd7fab1aa5e84a651b76b8903ed1d3cb93281"} Oct 30 17:29:37.479420 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:37.479344 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb" event={"ID":"1d6544b2-2ae4-4901-a710-f37d04215115","Type":"ContainerDied","Data":"a3fbed105c255c062388895346e305631c7ea1f9b99266c3a40992c79ce625d2"} Oct 30 17:29:37.479420 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:37.479374 4861 scope.go:122] "RemoveContainer" containerID="ef674785f9a9edd66d853f135dcbd7fab1aa5e84a651b76b8903ed1d3cb93281" Oct 30 17:29:37.489342 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:37.489323 4861 scope.go:122] "RemoveContainer" containerID="ef674785f9a9edd66d853f135dcbd7fab1aa5e84a651b76b8903ed1d3cb93281" Oct 30 17:29:37.489635 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:29:37.489614 4861 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef674785f9a9edd66d853f135dcbd7fab1aa5e84a651b76b8903ed1d3cb93281\": container with ID starting with ef674785f9a9edd66d853f135dcbd7fab1aa5e84a651b76b8903ed1d3cb93281 not found: ID does not exist" containerID="ef674785f9a9edd66d853f135dcbd7fab1aa5e84a651b76b8903ed1d3cb93281" Oct 30 17:29:37.489687 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:37.489643 4861 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef674785f9a9edd66d853f135dcbd7fab1aa5e84a651b76b8903ed1d3cb93281"} err="failed to get container status \"ef674785f9a9edd66d853f135dcbd7fab1aa5e84a651b76b8903ed1d3cb93281\": rpc error: code = NotFound desc = could not find container \"ef674785f9a9edd66d853f135dcbd7fab1aa5e84a651b76b8903ed1d3cb93281\": container with ID starting with ef674785f9a9edd66d853f135dcbd7fab1aa5e84a651b76b8903ed1d3cb93281 not found: ID does not exist" Oct 30 17:29:37.536255 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:37.536225 4861 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb"] Oct 30 17:29:37.549462 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:37.549439 4861 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-a77b48-7884d7d58c-bbzsb"] Oct 30 17:29:37.805866 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:37.805788 4861 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d6544b2-2ae4-4901-a710-f37d04215115" path="/var/lib/kubelet/pods/1d6544b2-2ae4-4901-a710-f37d04215115/volumes" Oct 30 17:29:48.214716 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:48.214673 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f4d877496-xgjn8_d8b4d873-2506-45c2-ad31-ed4b0b503ee8/manager/0.log" Oct 30 17:29:48.667449 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:48.667420 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5bzh6_c07163d1-a743-4cc0-9dbf-4b1f7809e578/manager/0.log" Oct 30 17:29:49.415753 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:49.415710 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6475ccc8c4-q7cpf_33810159-2c87-4168-8c6d-6dec7af9a368/manager/0.log" Oct 30 17:29:50.052650 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:50.052619 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-klj74_be3b019c-c736-4857-bda5-92df56b650c5/proxy/0.log" Oct 30 17:29:50.288126 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:29:50.288086 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:29:53.197251 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:29:53.197215 4861 scope.go:122] "RemoveContainer" containerID="17a2de0fa38cdd7949ff8caa121b3f3823de4b5db372f05d9b95d629d26d5db5" Oct 30 17:30:01.411499 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:01.411466 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f4d877496-xgjn8_d8b4d873-2506-45c2-ad31-ed4b0b503ee8/manager/0.log" Oct 30 17:30:01.847214 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:01.847186 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5bzh6_c07163d1-a743-4cc0-9dbf-4b1f7809e578/manager/0.log" Oct 30 17:30:02.586246 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:02.586214 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6475ccc8c4-q7cpf_33810159-2c87-4168-8c6d-6dec7af9a368/manager/0.log" Oct 30 17:30:03.203991 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:03.203940 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-klj74_be3b019c-c736-4857-bda5-92df56b650c5/proxy/0.log" Oct 30 17:30:03.920538 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:03.920503 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6f99ff7476-mb6nv_5f59b77e-b7c6-4e20-b138-cee7899675fc/maas-api/0.log" Oct 30 17:30:06.257054 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:06.257017 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-tsg47_ac065598-af1a-45c8-8720-00b2c12e4170/postgres/0.log" Oct 30 17:30:10.467569 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:10.467535 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ssrxb_c2cf11eb-a585-4fdf-bfad-b6e70b866976/discovery/0.log" Oct 30 17:30:10.876173 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:10.876092 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b54f85fcc-p7zbk_ac6c05a9-6522-473f-b77f-556378d2aa00/kube-auth-proxy/0.log" Oct 30 17:30:11.265357 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:11.265323 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-smq4m_f67a6d47-cf04-4307-af5b-ecb10e0612b1/istio-proxy/0.log" Oct 30 17:30:11.854570 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:11.854544 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zpxq6_a57f57e8-e81b-476f-8934-53a8cc63b937/payload-pre-processing/0.log" Oct 30 17:30:12.035334 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:12.035303 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gp6rg_a2319b60-ec05-4f37-b97a-79de3c482d58/payload-processing/0.log" Oct 30 17:30:12.235102 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:12.235077 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58c95ddf47-6m2jw_1ab95e94-bdff-4f1f-a0fe-095a2509d6f8/router/0.log" Oct 30 17:30:13.354341 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:13.354310 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-w9k67_c5c9cc23-e6a9-42e8-ae3f-f3b3cdd1eac2/main/0.log" Oct 30 17:30:13.550084 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:13.550058 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-q6hpd_b60d99a0-8ad6-4fc1-aaeb-8e77fe8e6471/main/0.log" Oct 30 17:30:13.933044 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:13.933010 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-w9pz8_ce7dda96-cefe-44b0-a922-3a1f706dd174/main/0.log" Oct 30 17:30:14.334558 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:14.334528 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-mkjhd_d37ba69c-1fa5-4f9b-a754-80057e2608b4/main/0.log" Oct 30 17:30:20.288344 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:30:20.288310 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:30:24.604179 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604145 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-skgpd/must-gather-9c79k"] Oct 30 17:30:24.604577 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604562 4861 cpu_manager.go:414] "containerMap: removing container" podUID="6f5e2f42-7122-4574-b519-891d32498176" containerName="payload-processing" Oct 30 17:30:24.604641 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604579 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5e2f42-7122-4574-b519-891d32498176" containerName="payload-processing" Oct 30 17:30:24.604641 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604592 4861 cpu_manager.go:414] "containerMap: removing container" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" Oct 30 17:30:24.604641 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604597 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" Oct 30 17:30:24.604641 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604604 4861 cpu_manager.go:414] "containerMap: removing container" podUID="29ca4500-8d2b-4e41-8ae2-13f452ded9c3" containerName="maas-api" Oct 30 17:30:24.604641 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604610 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ca4500-8d2b-4e41-8ae2-13f452ded9c3" containerName="maas-api" Oct 30 17:30:24.604641 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604619 4861 cpu_manager.go:414] "containerMap: removing container" podUID="cb294877-b8b7-45a5-b943-fc62473cedd7" containerName="payload-processing" Oct 30 17:30:24.604641 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604624 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb294877-b8b7-45a5-b943-fc62473cedd7" containerName="payload-processing" Oct 30 17:30:24.604641 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604629 4861 cpu_manager.go:414] "containerMap: removing container" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" Oct 30 17:30:24.604641 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604634 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" Oct 30 17:30:24.604641 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604643 4861 cpu_manager.go:414] "containerMap: removing container" podUID="cd5c0df9-6522-46f6-8222-117b7ad17bea" containerName="istio-proxy" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604648 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd5c0df9-6522-46f6-8222-117b7ad17bea" containerName="istio-proxy" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604656 4861 cpu_manager.go:414] "containerMap: removing container" podUID="1c826c68-020f-4496-8c43-a78ba51f4e9c" containerName="payload-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604660 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c826c68-020f-4496-8c43-a78ba51f4e9c" containerName="payload-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604665 4861 cpu_manager.go:414] "containerMap: removing container" podUID="f988c028-5967-4436-a8cd-30af1e920b40" containerName="maas-api" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604669 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f988c028-5967-4436-a8cd-30af1e920b40" containerName="maas-api" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604675 4861 cpu_manager.go:414] "containerMap: removing container" podUID="20eaed92-2c7a-4490-bc40-4efcb335b0bb" containerName="maas-api" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604682 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20eaed92-2c7a-4490-bc40-4efcb335b0bb" containerName="maas-api" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604695 4861 cpu_manager.go:414] "containerMap: removing container" podUID="4b8f2426-9e85-415e-9077-cf2471ad0875" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604704 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8f2426-9e85-415e-9077-cf2471ad0875" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604734 4861 cpu_manager.go:414] "containerMap: removing container" podUID="cff713aa-db97-4366-ae04-c2bb57261598" containerName="payload-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604742 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff713aa-db97-4366-ae04-c2bb57261598" containerName="payload-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604754 4861 cpu_manager.go:414] "containerMap: removing container" podUID="199a5f3f-241a-4d65-813c-928af105ac2b" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604760 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="199a5f3f-241a-4d65-813c-928af105ac2b" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604774 4861 cpu_manager.go:414] "containerMap: removing container" podUID="ce90a319-7468-4c37-8065-01f08642df9d" containerName="istio-proxy" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604781 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce90a319-7468-4c37-8065-01f08642df9d" containerName="istio-proxy" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604790 4861 cpu_manager.go:414] "containerMap: removing container" podUID="4ddd17b2-6142-4f07-97a6-8ac5db77cad3" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604795 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ddd17b2-6142-4f07-97a6-8ac5db77cad3" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604803 4861 cpu_manager.go:414] "containerMap: removing container" podUID="f7dd7b91-6cba-4446-9f47-c536520520bf" containerName="istio-proxy" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604807 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7dd7b91-6cba-4446-9f47-c536520520bf" containerName="istio-proxy" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604817 4861 cpu_manager.go:414] "containerMap: removing container" podUID="bb5c8a89-0cb6-4225-97c9-189984852da9" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604821 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5c8a89-0cb6-4225-97c9-189984852da9" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604829 4861 cpu_manager.go:414] "containerMap: removing container" podUID="3141d3a0-8e3a-4232-b9bc-2de718b19b1e" containerName="main" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604833 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3141d3a0-8e3a-4232-b9bc-2de718b19b1e" containerName="main" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604841 4861 cpu_manager.go:414] "containerMap: removing container" podUID="f3b08299-0d08-470e-ab89-d77ecd29a504" containerName="istio-proxy" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604846 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b08299-0d08-470e-ab89-d77ecd29a504" containerName="istio-proxy" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604852 4861 cpu_manager.go:414] "containerMap: removing container" podUID="a2c79376-2a02-4240-bde9-c58d20acb2f5" containerName="maas-api" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604856 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c79376-2a02-4240-bde9-c58d20acb2f5" containerName="maas-api" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604864 4861 cpu_manager.go:414] "containerMap: removing container" podUID="1d6544b2-2ae4-4901-a710-f37d04215115" containerName="maas-api" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604868 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d6544b2-2ae4-4901-a710-f37d04215115" containerName="maas-api" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604874 4861 cpu_manager.go:414] "containerMap: removing container" podUID="41e4e14f-88e4-4fac-a6fd-04baae40e066" containerName="istio-proxy" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604878 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e4e14f-88e4-4fac-a6fd-04baae40e066" containerName="istio-proxy" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604883 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b56c9a33-1b9e-48dc-9ebe-d80778b20e35" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604887 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b56c9a33-1b9e-48dc-9ebe-d80778b20e35" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604895 4861 cpu_manager.go:414] "containerMap: removing container" podUID="d980004b-323c-4ebf-be4e-d0356007ba80" containerName="payload-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604899 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d980004b-323c-4ebf-be4e-d0356007ba80" containerName="payload-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604906 4861 cpu_manager.go:414] "containerMap: removing container" podUID="932e0df6-6b3d-4904-a353-ae7faf5d2544" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604911 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932e0df6-6b3d-4904-a353-ae7faf5d2544" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604920 4861 cpu_manager.go:414] "containerMap: removing container" podUID="fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" containerName="istio-proxy" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604924 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" containerName="istio-proxy" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604932 4861 cpu_manager.go:414] "containerMap: removing container" podUID="04fa61ca-48d1-4fb0-99c9-2781775881d4" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604936 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fa61ca-48d1-4fb0-99c9-2781775881d4" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604942 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b1995227-e125-48f5-8bf8-ff46ae4fc16e" containerName="payload-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604946 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1995227-e125-48f5-8bf8-ff46ae4fc16e" containerName="payload-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604952 4861 cpu_manager.go:414] "containerMap: removing container" podUID="26c8b7b8-b899-4303-870f-dce7e002fb42" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604956 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c8b7b8-b899-4303-870f-dce7e002fb42" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604963 4861 cpu_manager.go:414] "containerMap: removing container" podUID="956234cd-9c32-48f5-853f-7646454ab761" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604966 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="956234cd-9c32-48f5-853f-7646454ab761" containerName="payload-pre-processing" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604972 4861 cpu_manager.go:414] "containerMap: removing container" podUID="7adb920d-0538-46bb-8c96-3d5f68902e50" containerName="istio-proxy" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604976 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7adb920d-0538-46bb-8c96-3d5f68902e50" containerName="istio-proxy" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604981 4861 cpu_manager.go:414] "containerMap: removing container" podUID="6254deaf-17a7-4214-928c-2d64d6da2684" containerName="maas-api" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604986 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6254deaf-17a7-4214-928c-2d64d6da2684" containerName="maas-api" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604992 4861 cpu_manager.go:414] "containerMap: removing container" podUID="63f181e5-e3d0-42e1-a509-b92d3658a9b5" containerName="maas-api" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.604996 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f181e5-e3d0-42e1-a509-b92d3658a9b5" containerName="maas-api" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605016 4861 cpu_manager.go:414] "containerMap: removing container" podUID="26f54ab0-924f-4d9d-aa69-69e9015b1bd5" containerName="istio-proxy" Oct 30 17:30:24.605000 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605022 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f54ab0-924f-4d9d-aa69-69e9015b1bd5" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605029 4861 cpu_manager.go:414] "containerMap: removing container" podUID="d252b794-0a34-4e07-94ee-2b1c7a2e40d0" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605034 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d252b794-0a34-4e07-94ee-2b1c7a2e40d0" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605044 4861 cpu_manager.go:414] "containerMap: removing container" podUID="71e9a70b-d593-49ba-82d4-0a022e09197a" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605049 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e9a70b-d593-49ba-82d4-0a022e09197a" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605056 4861 cpu_manager.go:414] "containerMap: removing container" podUID="6013ebf2-30f0-42a9-a31f-e66795df2b56" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605060 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6013ebf2-30f0-42a9-a31f-e66795df2b56" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605070 4861 cpu_manager.go:414] "containerMap: removing container" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605074 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605081 4861 cpu_manager.go:414] "containerMap: removing container" podUID="be98f9c8-1b20-40fa-84f5-74f69b2271c5" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605086 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be98f9c8-1b20-40fa-84f5-74f69b2271c5" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605092 4861 cpu_manager.go:414] "containerMap: removing container" podUID="1ad5ad27-a300-44f3-80af-ac24aabf8ebd" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605096 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ad5ad27-a300-44f3-80af-ac24aabf8ebd" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605104 4861 cpu_manager.go:414] "containerMap: removing container" podUID="40c65962-30bc-4ea9-b938-4a4917e875cc" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605108 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c65962-30bc-4ea9-b938-4a4917e875cc" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605114 4861 cpu_manager.go:414] "containerMap: removing container" podUID="d17e7f96-ac06-40a0-99dd-a486a538970b" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605117 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17e7f96-ac06-40a0-99dd-a486a538970b" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605124 4861 cpu_manager.go:414] "containerMap: removing container" podUID="24e3b70b-25df-4464-be5e-d5f77e1db8fd" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605128 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24e3b70b-25df-4464-be5e-d5f77e1db8fd" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605134 4861 cpu_manager.go:414] "containerMap: removing container" podUID="6e40b662-7953-4e26-921b-afb268c58d8e" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605138 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e40b662-7953-4e26-921b-afb268c58d8e" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605145 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b4ce5954-4591-4f16-84d5-36f11350dffc" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605149 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ce5954-4591-4f16-84d5-36f11350dffc" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605154 4861 cpu_manager.go:414] "containerMap: removing container" podUID="2087171b-22dc-444f-9f62-dd3003edbff9" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605158 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2087171b-22dc-444f-9f62-dd3003edbff9" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605163 4861 cpu_manager.go:414] "containerMap: removing container" podUID="276abb2e-f978-4d16-bfd4-c8838f173e31" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605168 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="276abb2e-f978-4d16-bfd4-c8838f173e31" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605173 4861 cpu_manager.go:414] "containerMap: removing container" podUID="9cfda264-9b06-4846-8245-394666f98198" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605177 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cfda264-9b06-4846-8245-394666f98198" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605186 4861 cpu_manager.go:414] "containerMap: removing container" podUID="3272339a-14a7-4af5-90cd-8beb09ec9e40" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605190 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3272339a-14a7-4af5-90cd-8beb09ec9e40" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605196 4861 cpu_manager.go:414] "containerMap: removing container" podUID="2f716c0d-9107-46e1-ae3c-7518ad0a62bb" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605200 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f716c0d-9107-46e1-ae3c-7518ad0a62bb" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605207 4861 cpu_manager.go:414] "containerMap: removing container" podUID="d7d2da07-ef07-4524-8627-6e2c5a201e6e" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605211 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d2da07-ef07-4524-8627-6e2c5a201e6e" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605219 4861 cpu_manager.go:414] "containerMap: removing container" podUID="0682eb07-3174-4f1f-af2c-a455e9ddc740" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605224 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0682eb07-3174-4f1f-af2c-a455e9ddc740" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605231 4861 cpu_manager.go:414] "containerMap: removing container" podUID="ef02b08b-f178-425d-86ce-0e72bdcb63a2" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605236 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef02b08b-f178-425d-86ce-0e72bdcb63a2" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605244 4861 cpu_manager.go:414] "containerMap: removing container" podUID="40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605248 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605254 4861 cpu_manager.go:414] "containerMap: removing container" podUID="8ed523f1-369d-4559-a4be-74633818bc17" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605258 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed523f1-369d-4559-a4be-74633818bc17" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605267 4861 cpu_manager.go:414] "containerMap: removing container" podUID="1f65a199-0838-4aa9-85d4-fb082370d14c" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605272 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f65a199-0838-4aa9-85d4-fb082370d14c" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605279 4861 cpu_manager.go:414] "containerMap: removing container" podUID="99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605283 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605289 4861 cpu_manager.go:414] "containerMap: removing container" podUID="caf4ec76-a9e6-487b-a504-67adbc815d25" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605294 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf4ec76-a9e6-487b-a504-67adbc815d25" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605301 4861 cpu_manager.go:414] "containerMap: removing container" podUID="9e67b1b9-0721-4914-85ed-6abf64720a33" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605305 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e67b1b9-0721-4914-85ed-6abf64720a33" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605310 4861 cpu_manager.go:414] "containerMap: removing container" podUID="f3b08299-0d08-470e-ab89-d77ecd29a504" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605314 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b08299-0d08-470e-ab89-d77ecd29a504" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605321 4861 cpu_manager.go:414] "containerMap: removing container" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605325 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605330 4861 cpu_manager.go:414] "containerMap: removing container" podUID="fa8794c5-5a39-47d8-b870-c7cf40526fb6" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605334 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa8794c5-5a39-47d8-b870-c7cf40526fb6" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605341 4861 cpu_manager.go:414] "containerMap: removing container" podUID="854e3a2c-0257-4f16-96ba-e86cc2ba282d" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605346 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="854e3a2c-0257-4f16-96ba-e86cc2ba282d" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605352 4861 cpu_manager.go:414] "containerMap: removing container" podUID="ce164706-e748-479e-a1a6-7977e983bc9e" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605357 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce164706-e748-479e-a1a6-7977e983bc9e" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605363 4861 cpu_manager.go:414] "containerMap: removing container" podUID="f0f77f41-bbe0-4095-8102-f685cc81e7e3" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605368 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f77f41-bbe0-4095-8102-f685cc81e7e3" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605374 4861 cpu_manager.go:414] "containerMap: removing container" podUID="3b0f1529-cc32-4603-987f-55b7d3a54c2a" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605378 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0f1529-cc32-4603-987f-55b7d3a54c2a" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605384 4861 cpu_manager.go:414] "containerMap: removing container" podUID="3c432eb4-2ac7-424b-9a55-6d21233b4807" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605389 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c432eb4-2ac7-424b-9a55-6d21233b4807" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605395 4861 cpu_manager.go:414] "containerMap: removing container" podUID="334f60fa-9e62-417d-8965-9a5a0f5bff47" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605400 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="334f60fa-9e62-417d-8965-9a5a0f5bff47" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605408 4861 cpu_manager.go:414] "containerMap: removing container" podUID="dda01ef9-7338-4724-ba7f-b9fe73cdfd4a" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605412 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda01ef9-7338-4724-ba7f-b9fe73cdfd4a" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605418 4861 cpu_manager.go:414] "containerMap: removing container" podUID="af41a080-c055-4886-8143-d586a24b1164" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605422 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af41a080-c055-4886-8143-d586a24b1164" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605427 4861 cpu_manager.go:414] "containerMap: removing container" podUID="1ce50453-ccab-4825-8ca9-b15579c1d0b4" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605431 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce50453-ccab-4825-8ca9-b15579c1d0b4" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605438 4861 cpu_manager.go:414] "containerMap: removing container" podUID="9345c3c1-50db-406b-81b6-202cb9b75496" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605442 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9345c3c1-50db-406b-81b6-202cb9b75496" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605448 4861 cpu_manager.go:414] "containerMap: removing container" podUID="0209596e-c5bc-4751-9864-0574cabf4197" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605452 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0209596e-c5bc-4751-9864-0574cabf4197" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605460 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b0891610-8a0f-41f6-937b-71b46b85defc" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605464 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0891610-8a0f-41f6-937b-71b46b85defc" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605470 4861 cpu_manager.go:414] "containerMap: removing container" podUID="3939645c-9b16-4a51-91a2-e5f043ddcf9f" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605474 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3939645c-9b16-4a51-91a2-e5f043ddcf9f" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605479 4861 cpu_manager.go:414] "containerMap: removing container" podUID="59476740-3ac5-4865-a330-505ee1c8cc0a" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605484 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59476740-3ac5-4865-a330-505ee1c8cc0a" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605491 4861 cpu_manager.go:414] "containerMap: removing container" podUID="20bdae97-71b8-4376-85b4-157138986bb2" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605495 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20bdae97-71b8-4376-85b4-157138986bb2" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605501 4861 cpu_manager.go:414] "containerMap: removing container" podUID="e1779c0b-52e6-4c97-b02c-116f12d11ff9" containerName="main" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605505 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1779c0b-52e6-4c97-b02c-116f12d11ff9" containerName="main" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605530 4861 cpu_manager.go:414] "containerMap: removing container" podUID="b9c6277c-c197-4b2a-945b-9c2553114762" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605535 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c6277c-c197-4b2a-945b-9c2553114762" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605541 4861 cpu_manager.go:414] "containerMap: removing container" podUID="0f099e88-ce47-485b-a7f6-2726c8f128c1" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605545 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f099e88-ce47-485b-a7f6-2726c8f128c1" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605552 4861 cpu_manager.go:414] "containerMap: removing container" podUID="2810edfd-dbe1-4bd4-9060-cf8c9e054741" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605556 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2810edfd-dbe1-4bd4-9060-cf8c9e054741" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605633 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c6277c-c197-4b2a-945b-9c2553114762" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605640 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f099e88-ce47-485b-a7f6-2726c8f128c1" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605646 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="2810edfd-dbe1-4bd4-9060-cf8c9e054741" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605656 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5e2f42-7122-4574-b519-891d32498176" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605663 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9a7f28a-b91a-4b4b-984d-a390a3d49776" containerName="main" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605668 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ca4500-8d2b-4e41-8ae2-13f452ded9c3" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605673 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb294877-b8b7-45a5-b943-fc62473cedd7" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605678 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66112b3-864b-46bf-a8dd-3f22c04f8d3f" containerName="main" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605684 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd5c0df9-6522-46f6-8222-117b7ad17bea" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605691 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c826c68-020f-4496-8c43-a78ba51f4e9c" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605696 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="f988c028-5967-4436-a8cd-30af1e920b40" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605700 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="20eaed92-2c7a-4490-bc40-4efcb335b0bb" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605707 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8f2426-9e85-415e-9077-cf2471ad0875" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605713 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff713aa-db97-4366-ae04-c2bb57261598" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605734 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="199a5f3f-241a-4d65-813c-928af105ac2b" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605744 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce90a319-7468-4c37-8065-01f08642df9d" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605754 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ddd17b2-6142-4f07-97a6-8ac5db77cad3" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605764 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce90a319-7468-4c37-8065-01f08642df9d" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605771 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7dd7b91-6cba-4446-9f47-c536520520bf" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605781 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5c8a89-0cb6-4225-97c9-189984852da9" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605787 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="3141d3a0-8e3a-4232-b9bc-2de718b19b1e" containerName="main" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605793 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b08299-0d08-470e-ab89-d77ecd29a504" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605799 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c79376-2a02-4240-bde9-c58d20acb2f5" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605804 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d6544b2-2ae4-4901-a710-f37d04215115" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605810 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e4e14f-88e4-4fac-a6fd-04baae40e066" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605815 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b56c9a33-1b9e-48dc-9ebe-d80778b20e35" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605825 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="d980004b-323c-4ebf-be4e-d0356007ba80" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605833 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="932e0df6-6b3d-4904-a353-ae7faf5d2544" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605846 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605851 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fa61ca-48d1-4fb0-99c9-2781775881d4" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605856 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1995227-e125-48f5-8bf8-ff46ae4fc16e" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605861 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c8b7b8-b899-4303-870f-dce7e002fb42" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605868 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="956234cd-9c32-48f5-853f-7646454ab761" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605873 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="7adb920d-0538-46bb-8c96-3d5f68902e50" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605878 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="6254deaf-17a7-4214-928c-2d64d6da2684" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605884 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f181e5-e3d0-42e1-a509-b92d3658a9b5" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605889 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f54ab0-924f-4d9d-aa69-69e9015b1bd5" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605894 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="d252b794-0a34-4e07-94ee-2b1c7a2e40d0" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605899 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e9a70b-d593-49ba-82d4-0a022e09197a" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605904 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="6013ebf2-30f0-42a9-a31f-e66795df2b56" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605910 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ce5954-4591-4f16-84d5-36f11350dffc" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605917 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="1837e2a6-812b-4200-bee2-21c23c1a8bb9" containerName="main" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605923 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="be98f9c8-1b20-40fa-84f5-74f69b2271c5" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605928 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ad5ad27-a300-44f3-80af-ac24aabf8ebd" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605934 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c65962-30bc-4ea9-b938-4a4917e875cc" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605940 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17e7f96-ac06-40a0-99dd-a486a538970b" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605945 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="24e3b70b-25df-4464-be5e-d5f77e1db8fd" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605950 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="2087171b-22dc-444f-9f62-dd3003edbff9" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605955 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="276abb2e-f978-4d16-bfd4-c8838f173e31" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605960 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cfda264-9b06-4846-8245-394666f98198" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605965 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="3272339a-14a7-4af5-90cd-8beb09ec9e40" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605970 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f716c0d-9107-46e1-ae3c-7518ad0a62bb" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605975 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d2da07-ef07-4524-8627-6e2c5a201e6e" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605980 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="0682eb07-3174-4f1f-af2c-a455e9ddc740" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605986 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e40b662-7953-4e26-921b-afb268c58d8e" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605991 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef02b08b-f178-425d-86ce-0e72bdcb63a2" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.605996 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d5d14f-0f64-4d4c-a6fa-cfe32c5a7ddf" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606000 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed523f1-369d-4559-a4be-74633818bc17" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606005 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f65a199-0838-4aa9-85d4-fb082370d14c" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606010 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ba73dc-2ca3-4c9d-a839-f0a7bd9ec1d6" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606015 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf4ec76-a9e6-487b-a504-67adbc815d25" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606020 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e67b1b9-0721-4914-85ed-6abf64720a33" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606025 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b08299-0d08-470e-ab89-d77ecd29a504" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606031 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2d86a2-ddd1-4070-9b52-f6b05d9e38bc" containerName="main" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606036 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa8794c5-5a39-47d8-b870-c7cf40526fb6" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606040 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="854e3a2c-0257-4f16-96ba-e86cc2ba282d" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606045 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce164706-e748-479e-a1a6-7977e983bc9e" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606050 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f77f41-bbe0-4095-8102-f685cc81e7e3" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606055 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0f1529-cc32-4603-987f-55b7d3a54c2a" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606061 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c432eb4-2ac7-424b-9a55-6d21233b4807" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606065 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="334f60fa-9e62-417d-8965-9a5a0f5bff47" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606071 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda01ef9-7338-4724-ba7f-b9fe73cdfd4a" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606075 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="af41a080-c055-4886-8143-d586a24b1164" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606080 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce50453-ccab-4825-8ca9-b15579c1d0b4" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606085 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="9345c3c1-50db-406b-81b6-202cb9b75496" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606091 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="0209596e-c5bc-4751-9864-0574cabf4197" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606097 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0891610-8a0f-41f6-937b-71b46b85defc" containerName="maas-api" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606103 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="3939645c-9b16-4a51-91a2-e5f043ddcf9f" containerName="payload-pre-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606108 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="59476740-3ac5-4865-a330-505ee1c8cc0a" containerName="payload-processing" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606113 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="20bdae97-71b8-4376-85b4-157138986bb2" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606118 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1779c0b-52e6-4c97-b02c-116f12d11ff9" containerName="main" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606203 4861 cpu_manager.go:414] "containerMap: removing container" podUID="ce90a319-7468-4c37-8065-01f08642df9d" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606209 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce90a319-7468-4c37-8065-01f08642df9d" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606216 4861 cpu_manager.go:414] "containerMap: removing container" podUID="f7dd7b91-6cba-4446-9f47-c536520520bf" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606221 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7dd7b91-6cba-4446-9f47-c536520520bf" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606232 4861 cpu_manager.go:414] "containerMap: removing container" podUID="41e4e14f-88e4-4fac-a6fd-04baae40e066" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606236 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e4e14f-88e4-4fac-a6fd-04baae40e066" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606246 4861 cpu_manager.go:414] "containerMap: removing container" podUID="fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606250 4861 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606310 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe89fc65-9620-4dc6-98da-cd1a3d57dbc3" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606322 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7dd7b91-6cba-4446-9f47-c536520520bf" containerName="istio-proxy" Oct 30 17:30:24.606792 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.606328 4861 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e4e14f-88e4-4fac-a6fd-04baae40e066" containerName="istio-proxy" Oct 30 17:30:24.613777 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.609664 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skgpd/must-gather-9c79k" Oct 30 17:30:24.630402 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.630375 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skgpd/must-gather-9c79k"] Oct 30 17:30:24.678573 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.678539 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skkgt\" (UniqueName: \"kubernetes.io/projected/ae7581df-f1da-44d2-aa24-a1330ece62c5-kube-api-access-skkgt\") pod \"must-gather-9c79k\" (UID: \"ae7581df-f1da-44d2-aa24-a1330ece62c5\") " pod="openshift-must-gather-skgpd/must-gather-9c79k" Oct 30 17:30:24.678791 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.678600 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ae7581df-f1da-44d2-aa24-a1330ece62c5-must-gather-output\") pod \"must-gather-9c79k\" (UID: \"ae7581df-f1da-44d2-aa24-a1330ece62c5\") " pod="openshift-must-gather-skgpd/must-gather-9c79k" Oct 30 17:30:24.685325 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.685294 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-skgpd\"/\"openshift-service-ca.crt\"" Oct 30 17:30:24.685462 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.685363 4861 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-skgpd\"/\"kube-root-ca.crt\"" Oct 30 17:30:24.728067 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.728046 4861 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-skgpd\"/\"default-dockercfg-fv925\"" Oct 30 17:30:24.779698 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.779663 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skkgt\" (UniqueName: \"kubernetes.io/projected/ae7581df-f1da-44d2-aa24-a1330ece62c5-kube-api-access-skkgt\") pod \"must-gather-9c79k\" (UID: \"ae7581df-f1da-44d2-aa24-a1330ece62c5\") " pod="openshift-must-gather-skgpd/must-gather-9c79k" Oct 30 17:30:24.779867 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.779747 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ae7581df-f1da-44d2-aa24-a1330ece62c5-must-gather-output\") pod \"must-gather-9c79k\" (UID: \"ae7581df-f1da-44d2-aa24-a1330ece62c5\") " pod="openshift-must-gather-skgpd/must-gather-9c79k" Oct 30 17:30:24.780094 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.780074 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ae7581df-f1da-44d2-aa24-a1330ece62c5-must-gather-output\") pod \"must-gather-9c79k\" (UID: \"ae7581df-f1da-44d2-aa24-a1330ece62c5\") " pod="openshift-must-gather-skgpd/must-gather-9c79k" Oct 30 17:30:24.806110 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.806080 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skkgt\" (UniqueName: \"kubernetes.io/projected/ae7581df-f1da-44d2-aa24-a1330ece62c5-kube-api-access-skkgt\") pod \"must-gather-9c79k\" (UID: \"ae7581df-f1da-44d2-aa24-a1330ece62c5\") " pod="openshift-must-gather-skgpd/must-gather-9c79k" Oct 30 17:30:24.923928 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:24.923851 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skgpd/must-gather-9c79k" Oct 30 17:30:25.225687 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:25.225634 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skgpd/must-gather-9c79k"] Oct 30 17:30:25.228135 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:30:25.228108 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae7581df_f1da_44d2_aa24_a1330ece62c5.slice/crio-a58bc93e48e1021be1c457a1e42fb551a0a9189ce4676502e954f9a55d4e5269 WatchSource:0}: Error finding container a58bc93e48e1021be1c457a1e42fb551a0a9189ce4676502e954f9a55d4e5269: Status 404 returned error can't find the container with id a58bc93e48e1021be1c457a1e42fb551a0a9189ce4676502e954f9a55d4e5269 Oct 30 17:30:25.230245 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:25.230225 4861 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:30:25.715436 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:25.715356 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skgpd/must-gather-9c79k" event={"ID":"ae7581df-f1da-44d2-aa24-a1330ece62c5","Type":"ContainerStarted","Data":"a58bc93e48e1021be1c457a1e42fb551a0a9189ce4676502e954f9a55d4e5269"} Oct 30 17:30:26.723483 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:26.723437 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skgpd/must-gather-9c79k" event={"ID":"ae7581df-f1da-44d2-aa24-a1330ece62c5","Type":"ContainerStarted","Data":"49df70ab837c70a52ef845f3b0c8d87274f683906844dd2f6d1705f52ef1ddcb"} Oct 30 17:30:26.723935 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:26.723491 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skgpd/must-gather-9c79k" event={"ID":"ae7581df-f1da-44d2-aa24-a1330ece62c5","Type":"ContainerStarted","Data":"dff73582ebf21d48090a33f68039373c9696b12c80a86eada0a25eefab416dcb"} Oct 30 17:30:26.769781 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:26.769695 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-skgpd/must-gather-9c79k" podStartSLOduration=1.9774441139999999 podStartE2EDuration="2.769675367s" totalImagesPullingTime="792.231253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:26.768599513 +0000 UTC m=+4057.111273551" watchObservedRunningTime="2026-10-30 17:30:26.769675367 +0000 UTC m=+4057.112349406" Oct 30 17:30:29.273111 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:29.273071 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pq64k_cb9bd726-ba59-4c55-838f-e00276ca9bef/global-pull-secret-syncer/0.log" Oct 30 17:30:29.633133 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:29.633053 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-2sc7w_49aa267f-cf95-4d9b-98c4-df106fcd9620/connection-checker/0.log" Oct 30 17:30:29.753657 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:29.753629 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5r74t_1026393f-ba8e-408a-b7df-a1d03e047b1f/connection-checker/0.log" Oct 30 17:30:29.900835 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:29.898134 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-9b9bn_c8ddcddb-5053-4950-bc4e-5d8f83ec8f08/connection-checker/0.log" Oct 30 17:30:30.146376 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:30.146344 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gjhv5_6089204f-1338-4875-b311-2fa1b27a03d5/konnectivity-agent/0.log" Oct 30 17:30:30.586741 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:30.582428 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-179.ec2.internal_26f3b181327243d25a7eb8229eca8f4d/haproxy/0.log" Oct 30 17:30:32.387653 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:32.387610 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f4d877496-xgjn8_d8b4d873-2506-45c2-ad31-ed4b0b503ee8/manager/0.log" Oct 30 17:30:32.447898 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:32.447870 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f4d877496-xgjn8_d8b4d873-2506-45c2-ad31-ed4b0b503ee8/manager/0.log" Oct 30 17:30:32.644783 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:32.644624 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5bzh6_c07163d1-a743-4cc0-9dbf-4b1f7809e578/manager/0.log" Oct 30 17:30:32.771969 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:32.768622 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5bzh6_c07163d1-a743-4cc0-9dbf-4b1f7809e578/manager/0.log" Oct 30 17:30:35.043857 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:35.043821 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6475ccc8c4-q7cpf_33810159-2c87-4168-8c6d-6dec7af9a368/manager/0.log" Oct 30 17:30:35.221260 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:35.221228 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6475ccc8c4-q7cpf_33810159-2c87-4168-8c6d-6dec7af9a368/manager/0.log" Oct 30 17:30:39.237123 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.237090 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-klj74_be3b019c-c736-4857-bda5-92df56b650c5/proxy/0.log" Oct 30 17:30:39.443080 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.443022 4861 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662"] Oct 30 17:30:39.448061 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.448038 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:39.469996 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.469964 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662"] Oct 30 17:30:39.509861 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.509777 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-klj74_be3b019c-c736-4857-bda5-92df56b650c5/proxy/0.log" Oct 30 17:30:39.582276 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.582235 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04394669-72c9-4a04-8203-eb5f9984d3c9-lib-modules\") pod \"perf-node-gather-daemonset-gv662\" (UID: \"04394669-72c9-4a04-8203-eb5f9984d3c9\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:39.582564 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.582544 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04394669-72c9-4a04-8203-eb5f9984d3c9-podres\") pod \"perf-node-gather-daemonset-gv662\" (UID: \"04394669-72c9-4a04-8203-eb5f9984d3c9\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:39.585859 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.582789 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc7ph\" (UniqueName: \"kubernetes.io/projected/04394669-72c9-4a04-8203-eb5f9984d3c9-kube-api-access-bc7ph\") pod \"perf-node-gather-daemonset-gv662\" (UID: \"04394669-72c9-4a04-8203-eb5f9984d3c9\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:39.585859 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.582884 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04394669-72c9-4a04-8203-eb5f9984d3c9-sys\") pod \"perf-node-gather-daemonset-gv662\" (UID: \"04394669-72c9-4a04-8203-eb5f9984d3c9\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:39.585859 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.582915 4861 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04394669-72c9-4a04-8203-eb5f9984d3c9-proc\") pod \"perf-node-gather-daemonset-gv662\" (UID: \"04394669-72c9-4a04-8203-eb5f9984d3c9\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:39.683472 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.683434 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04394669-72c9-4a04-8203-eb5f9984d3c9-sys\") pod \"perf-node-gather-daemonset-gv662\" (UID: \"04394669-72c9-4a04-8203-eb5f9984d3c9\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:39.683668 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.683478 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04394669-72c9-4a04-8203-eb5f9984d3c9-proc\") pod \"perf-node-gather-daemonset-gv662\" (UID: \"04394669-72c9-4a04-8203-eb5f9984d3c9\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:39.683668 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.683520 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04394669-72c9-4a04-8203-eb5f9984d3c9-lib-modules\") pod \"perf-node-gather-daemonset-gv662\" (UID: \"04394669-72c9-4a04-8203-eb5f9984d3c9\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:39.683668 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.683559 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04394669-72c9-4a04-8203-eb5f9984d3c9-sys\") pod \"perf-node-gather-daemonset-gv662\" (UID: \"04394669-72c9-4a04-8203-eb5f9984d3c9\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:39.683668 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.683567 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04394669-72c9-4a04-8203-eb5f9984d3c9-podres\") pod \"perf-node-gather-daemonset-gv662\" (UID: \"04394669-72c9-4a04-8203-eb5f9984d3c9\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:39.683668 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.683627 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/04394669-72c9-4a04-8203-eb5f9984d3c9-proc\") pod \"perf-node-gather-daemonset-gv662\" (UID: \"04394669-72c9-4a04-8203-eb5f9984d3c9\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:39.683668 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.683646 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/04394669-72c9-4a04-8203-eb5f9984d3c9-lib-modules\") pod \"perf-node-gather-daemonset-gv662\" (UID: \"04394669-72c9-4a04-8203-eb5f9984d3c9\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:39.683668 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.683661 4861 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc7ph\" (UniqueName: \"kubernetes.io/projected/04394669-72c9-4a04-8203-eb5f9984d3c9-kube-api-access-bc7ph\") pod \"perf-node-gather-daemonset-gv662\" (UID: \"04394669-72c9-4a04-8203-eb5f9984d3c9\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:39.684137 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.683690 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/04394669-72c9-4a04-8203-eb5f9984d3c9-podres\") pod \"perf-node-gather-daemonset-gv662\" (UID: \"04394669-72c9-4a04-8203-eb5f9984d3c9\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:39.708385 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.708351 4861 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc7ph\" (UniqueName: \"kubernetes.io/projected/04394669-72c9-4a04-8203-eb5f9984d3c9-kube-api-access-bc7ph\") pod \"perf-node-gather-daemonset-gv662\" (UID: \"04394669-72c9-4a04-8203-eb5f9984d3c9\") " pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:39.766171 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:39.766086 4861 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:40.085004 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:40.084926 4861 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662"] Oct 30 17:30:40.087533 ip-10-0-131-179 kubenswrapper[4861]: W1030 17:30:40.087505 4861 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod04394669_72c9_4a04_8203_eb5f9984d3c9.slice/crio-07da8f8fbbe999028f856fc71ae5e543052e2f45b89e3d0bec7cd4500362aaac WatchSource:0}: Error finding container 07da8f8fbbe999028f856fc71ae5e543052e2f45b89e3d0bec7cd4500362aaac: Status 404 returned error can't find the container with id 07da8f8fbbe999028f856fc71ae5e543052e2f45b89e3d0bec7cd4500362aaac Oct 30 17:30:40.681215 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:40.681132 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f4d877496-xgjn8_d8b4d873-2506-45c2-ad31-ed4b0b503ee8/manager/0.log" Oct 30 17:30:40.789105 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:40.789071 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f4d877496-xgjn8_d8b4d873-2506-45c2-ad31-ed4b0b503ee8/manager/0.log" Oct 30 17:30:40.831676 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:40.831646 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5bzh6_c07163d1-a743-4cc0-9dbf-4b1f7809e578/manager/0.log" Oct 30 17:30:40.860024 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:40.859991 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" event={"ID":"04394669-72c9-4a04-8203-eb5f9984d3c9","Type":"ContainerStarted","Data":"d82beb56aeb23042d7707226bfb0df73cb77ebd9164d737b939a245843bcd2a0"} Oct 30 17:30:40.860198 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:40.860049 4861 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" event={"ID":"04394669-72c9-4a04-8203-eb5f9984d3c9","Type":"ContainerStarted","Data":"07da8f8fbbe999028f856fc71ae5e543052e2f45b89e3d0bec7cd4500362aaac"} Oct 30 17:30:40.860198 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:40.860115 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:40.904860 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:40.904812 4861 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" podStartSLOduration=1.904795778 podStartE2EDuration="1.904795778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:30:40.903714542 +0000 UTC m=+4071.246388581" watchObservedRunningTime="2026-10-30 17:30:40.904795778 +0000 UTC m=+4071.247469860" Oct 30 17:30:40.935717 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:40.935659 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5bzh6_c07163d1-a743-4cc0-9dbf-4b1f7809e578/manager/0.log" Oct 30 17:30:43.041970 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:43.041914 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6475ccc8c4-q7cpf_33810159-2c87-4168-8c6d-6dec7af9a368/manager/0.log" Oct 30 17:30:43.067791 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:43.067760 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-gn2hp_93b8a9b8-43df-4dc0-97da-4bfd11f373cd/cluster-monitoring-operator/0.log" Oct 30 17:30:43.131329 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:43.131299 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6475ccc8c4-q7cpf_33810159-2c87-4168-8c6d-6dec7af9a368/manager/0.log" Oct 30 17:30:44.179238 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:44.179202 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g57lb_97c994bf-ac26-438e-86c0-012c237ae965/node-exporter/0.log" Oct 30 17:30:44.299878 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:44.299853 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g57lb_97c994bf-ac26-438e-86c0-012c237ae965/kube-rbac-proxy/0.log" Oct 30 17:30:44.421422 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:44.421339 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g57lb_97c994bf-ac26-438e-86c0-012c237ae965/init-textfile/0.log" Oct 30 17:30:46.877339 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:46.877310 4861 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-skgpd/perf-node-gather-daemonset-gv662" Oct 30 17:30:47.010917 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:47.010891 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-klj74_be3b019c-c736-4857-bda5-92df56b650c5/proxy/0.log" Oct 30 17:30:47.616803 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:47.616770 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-klj74_be3b019c-c736-4857-bda5-92df56b650c5/proxy/0.log" Oct 30 17:30:48.413496 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:48.413469 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f4d877496-xgjn8_d8b4d873-2506-45c2-ad31-ed4b0b503ee8/manager/0.log" Oct 30 17:30:48.564098 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:48.564051 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5bzh6_c07163d1-a743-4cc0-9dbf-4b1f7809e578/manager/0.log" Oct 30 17:30:48.914481 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:48.914451 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f4d877496-xgjn8_d8b4d873-2506-45c2-ad31-ed4b0b503ee8/manager/0.log" Oct 30 17:30:49.057021 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:49.056989 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-5bzh6_c07163d1-a743-4cc0-9dbf-4b1f7809e578/manager/0.log" Oct 30 17:30:49.352477 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:49.352436 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-6xx9g_78ae398a-fdb1-48f8-94fa-51eb582e663a/networking-console-plugin/0.log" Oct 30 17:30:50.306457 ip-10-0-131-179 kubenswrapper[4861]: E1030 17:30:50.306408 4861 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wnmqt" containerName="extractor" Oct 30 17:30:50.579175 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:50.579066 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6475ccc8c4-q7cpf_33810159-2c87-4168-8c6d-6dec7af9a368/manager/0.log" Oct 30 17:30:50.703987 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:50.703953 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/2.log" Oct 30 17:30:50.710409 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:50.710377 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-szx5s_5a25e3b8-4f22-4fdb-86a9-70d1c594236d/console-operator/3.log" Oct 30 17:30:50.912227 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:50.912134 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6475ccc8c4-q7cpf_33810159-2c87-4168-8c6d-6dec7af9a368/manager/0.log" Oct 30 17:30:52.808344 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:52.808306 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ssrxb_c2cf11eb-a585-4fdf-bfad-b6e70b866976/discovery/0.log" Oct 30 17:30:52.926782 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:52.926756 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b54f85fcc-p7zbk_ac6c05a9-6522-473f-b77f-556378d2aa00/kube-auth-proxy/0.log" Oct 30 17:30:53.060033 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:53.059961 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-smq4m_f67a6d47-cf04-4307-af5b-ecb10e0612b1/istio-proxy/0.log" Oct 30 17:30:53.085230 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:53.085184 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-rmgmf_a21d3561-30d9-4a48-816b-5b7be4e5813d/volume-data-source-validator/0.log" Oct 30 17:30:53.190734 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:53.190703 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zpxq6_a57f57e8-e81b-476f-8934-53a8cc63b937/payload-pre-processing/0.log" Oct 30 17:30:53.257765 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:53.257736 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gp6rg_a2319b60-ec05-4f37-b97a-79de3c482d58/payload-processing/0.log" Oct 30 17:30:53.329013 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:53.328946 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58c95ddf47-6m2jw_1ab95e94-bdff-4f1f-a0fe-095a2509d6f8/router/0.log" Oct 30 17:30:54.751821 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:54.751787 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-klj74_be3b019c-c736-4857-bda5-92df56b650c5/proxy/0.log" Oct 30 17:30:54.885159 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:54.885128 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-klj74_be3b019c-c736-4857-bda5-92df56b650c5/proxy/0.log" Oct 30 17:30:55.356060 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:55.356029 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-82w6j_f79b5c8d-c736-4db0-93b4-b6a40ad58955/dns/0.log" Oct 30 17:30:55.477104 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:55.477047 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-82w6j_f79b5c8d-c736-4db0-93b4-b6a40ad58955/kube-rbac-proxy/0.log" Oct 30 17:30:56.227187 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:56.227153 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6xn9h_1d0df1a2-2140-46d5-8662-1148296fb5a8/dns-node-resolver/0.log" Oct 30 17:30:56.501466 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:56.501382 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ssrxb_c2cf11eb-a585-4fdf-bfad-b6e70b866976/discovery/0.log" Oct 30 17:30:56.596351 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:56.596321 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b54f85fcc-p7zbk_ac6c05a9-6522-473f-b77f-556378d2aa00/kube-auth-proxy/0.log" Oct 30 17:30:56.635333 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:56.635301 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ssrxb_c2cf11eb-a585-4fdf-bfad-b6e70b866976/discovery/0.log" Oct 30 17:30:56.718569 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:56.718533 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-smq4m_f67a6d47-cf04-4307-af5b-ecb10e0612b1/istio-proxy/0.log" Oct 30 17:30:56.776537 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:56.776471 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b54f85fcc-p7zbk_ac6c05a9-6522-473f-b77f-556378d2aa00/kube-auth-proxy/0.log" Oct 30 17:30:56.835888 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:56.835839 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zpxq6_a57f57e8-e81b-476f-8934-53a8cc63b937/payload-pre-processing/0.log" Oct 30 17:30:56.893656 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:56.893629 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gp6rg_a2319b60-ec05-4f37-b97a-79de3c482d58/payload-processing/0.log" Oct 30 17:30:56.899568 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:56.899542 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-smq4m_f67a6d47-cf04-4307-af5b-ecb10e0612b1/istio-proxy/0.log" Oct 30 17:30:56.960845 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:56.960797 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58c95ddf47-6m2jw_1ab95e94-bdff-4f1f-a0fe-095a2509d6f8/router/0.log" Oct 30 17:30:57.018007 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:57.017978 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zpxq6_a57f57e8-e81b-476f-8934-53a8cc63b937/payload-pre-processing/0.log" Oct 30 17:30:57.075696 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:57.075615 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gp6rg_a2319b60-ec05-4f37-b97a-79de3c482d58/payload-processing/0.log" Oct 30 17:30:57.141976 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:57.141945 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58c95ddf47-6m2jw_1ab95e94-bdff-4f1f-a0fe-095a2509d6f8/router/0.log" Oct 30 17:30:57.672921 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:57.672887 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-ssrxb_c2cf11eb-a585-4fdf-bfad-b6e70b866976/discovery/0.log" Oct 30 17:30:57.768980 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:57.768954 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b54f85fcc-p7zbk_ac6c05a9-6522-473f-b77f-556378d2aa00/kube-auth-proxy/0.log" Oct 30 17:30:57.850132 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:57.850102 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pbsvn_9ab0447a-bdef-45aa-92c3-e9d35ee9c2be/node-ca/0.log" Oct 30 17:30:57.902031 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:57.901998 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7854949c9-smq4m_f67a6d47-cf04-4307-af5b-ecb10e0612b1/istio-proxy/0.log" Oct 30 17:30:58.033770 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:58.031567 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-zpxq6_a57f57e8-e81b-476f-8934-53a8cc63b937/payload-pre-processing/0.log" Oct 30 17:30:58.094491 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:58.094470 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-gp6rg_a2319b60-ec05-4f37-b97a-79de3c482d58/payload-processing/0.log" Oct 30 17:30:58.178092 ip-10-0-131-179 kubenswrapper[4861]: I1030 17:30:58.178066 4861 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-58c95ddf47-6m2jw_1ab95e94-bdff-4f1f-a0fe-095a2509d6f8/router/0.log"